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ABSTRACT 


One of the most significant challenges in large-scale climate modeling, as well as in high-performance 
computing in other scientific fields, is that of effectively integrating many software models from multiple 
contributors. A software framework facilitates the integration task, both in the development and runtime 
stages of the simulation. Effective software frameworks reduce the programming burden lor the investigators, 
freeing them to focus more on the science and less on the parallel communication implementation, while 
maintaining high performance across numerous supercomputer and workstation architectures. 

This document surveys numerous software frameworks for potential use in Earth science modeling. Several 
frameworks arc evaluated in depth, including Parallel Object-Oriented Methods and Applications (POOMA), 
Cactus (from the relativistic physics community), Overture, Goddard Earth Modeling System (GEMS), the 
National Center for Atmospheric Research Flux Coupler, and UCLA/UCB Distributed Data Broker (DDB ). 
Frameworks evaluated in less detail include ROOT, Parallel Application Workspace (PAWS), and Advanced 
Large-Scale Integrated Computational Environment (ALICE). A host of other frameworks and related tools 
are referenced in this context. The frameworks are evaluated individually and also compared with each other. 
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1.0 INTRODUCTION 


This report surveys existing modeling frameworks to provide insight into available software that could 
potentially support the development of a community Earth System Modeling Framework (ESMF). The 
survey focused on the six frameworks listed below; however, many more were examined in less detail. 

• Parallel Object-Oriented Methods and Applications (POOMA) 

• Cactus 

• Overture (Object-Oriented Tools for Solving CFD and Combustion Problems in Complex Moving 
Geometry) 

• Goddard Earth Modeling System (GEMS) 

• National Center for Atmospheric Research (NCAR) Flux Coupler 

• University of California, Los Angeles, (UCLA) Distributed Data Broker (DDB) 

A Web site has been developed [ 1 ] to provide reference to the investigated frameworks, as well as others that 
are not mentioned in this report. In addition, an interim brieting of the survey results is provided at that site. 


This report is a focused rather than a general survey in that it reviews the frameworks in the context of ESMF 
needs rather than in the context of the specific problem the framework was originally designed to solve. The 
ESMF needs are expressed in both the Cooperative Agreement Notice (CAN) [2] and in documents arising 
from meetings of the Common Modeling Infrastructure working group (CMIWG) [3]. These needs arc 
informally documented here. Framework deficiencies relative to ESMF needs should not be considered 
general weaknesses, since none of the frameworks were specifically developed to address those needs. 

The report is divided into three major sections: Introduction, Framework Survey, and Discussions and 
Recommendations. 

Section 2, Framework Survey, which follows this Introduction, is divided into eight major subsections. 
Subsection 2.1 provides the information collection approach and the major points-of-contact (POC’s) within 
the organizations that developed the framework. Subsections 2.2 through 2.7 provide survey results lor the 
six primary frameworks (listed above). Each subsection is further divided into five sections: introduction, 
synopsis, description, evaluation, and references. Separate references lists are provided lor each lramework to 
simplify the search for the reader and because there is little overlap. This approach is used throughout the 
document. Subsection 2.8 provides less detailed information on three other frameworks — ROOT, Parallel 
Application Workspace (PAWS), and Advanced Large-Scale Integrated Computational Environment 
(ALICE), that may be of interest to the climate community but which were not reviewed at the same level of 
detail as the others. As indicated earlier, an extensive set of links to these and other frameworks identified in 
the survey may be found at the survey web site [ 1 ]. 

Section 3, Discussion and Recommendations, is divided in four subsections. Subsection 3.1 is entitled 
“Motivation for ESMF.” This subsection presents results of reviewing the CAN and findings of CMIWG to 
identify the underlying requirements of a community ESMF. Subsection 3.2 looks at the relative capabilities 
of the surveyed frameworks relative to ESMF requirements. Section 3.3 provides recommendations on the 
types of framework solutions that may be possible to satisfy ESMF needs. Finally, Section 3.4 provides the 
references for all of Section 3. 
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1.1 References 


1 Survey results. URL: http://esdcd.gsrc. nasa.gov/ESS/esmr tasc/i ndex.htm I 

Stall, National Aeronautics and Space Administration (NASA), 2000: NASA High Performance 
Computing and Communications (HPCC)/Earth and Space Sciences (ESS) Cooperative Agreement Notice 
(CAN) lor Solicitation of Round-3 Grand Challenge Investigations: Increasing Interoperability and 
Perlormance ol Grand Challenge Applications in the Earth, Space, Life, and Microgravity Sciences. 

URL: http://carth.nasa.gov/nra/currcnt/can00ocs01/ 

CMIWCi. hllp://janus. gsfc.nasa.gov/-mkistler/inrra/master.html 
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2.0 FRAMEWORK SURVEY 


2.1 Information Collection 

A variety of sources was used to obtain information about each framework. These sources included Web 
information, technical documentation (e.g., technical reports, sol t ware documents), soltwarc source code 
review 7 , and personal communication with the framework developers. The personal communications were 
particularly useful and most organizations were gracious enough to provide more than one telephone 
interview to clarify findings. In the early stages of the survey, a list of basic inquiry areas was formulated to 
focus the information search. This was done to ensure that similar information was collected from each 
organization, although it did not necessarily limit the framework review. These inquiry areas included the 
following: 

• Requirements/capabilities — Major goals of the framework, current capabilities, parallel 
implementation, unique capabilities, luturc capabilities (i.c., are important new features planned?). 

• Language — Implementation language of the framework and language requirements of applications 
using the framework. 

• High-level design — Highest level design in terms of a hierarchical structure diagram or functional 
diagram. 

• Tools and utilities included — Libraries, key Fortran subroutines, etc., that might be of interest to the 
climate community (e.g., grid utilities, parallel communications utilities). 

• Application interface — How would an application interface with the framework? 

• Other issues — Maturity of framework (number of users, number of versions issued, number of 
developers on team), performance optimization (optimization efforts by developers to improve 
performance), etc. 

The points-of-contact {POC’s) for the six frameworks are shown in table 1 . 


Table 1. POC’s for the Six Surveyed Frameworks 


Framework 

Organization 

Primary POC’s 

POOMA 

Los Alamos National Laboratory 

Julian Cummings 

Cactus 

Collaborative effort within the 
relativistic physics community 

Ed Siedel 

Overture 

Lawrence Livermore National 
Laboratory 

David Brown, Bill Henshaw, Dan Quinlan 

GEMS 

NASA NSIPP and DAO 

Max Suarez 

Flux Coupler 

NCAR 

Brian Kauffman and Tom Bettge 

Distributed Data Broke 

UCLA Atmospheric Sciences Department 

Tony Drummond 


The results of the survey arc provided in the remaining subsections. Subsection 2.2 describes POOMA, 2.3 
describes Cactus, 2.4 describes Overture, 2.5 describes GEMS, 2.6 describes NCAR Flux Coupler, 2.7 
describes the UCLA Distributed Data Broker, and 2.8 describes ROOT, PAWS and ALICE. 
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2.2 POOMA 


2.2.1 Introduction 

Parallel Object-Oriented Methods and Applications (POOMA) originates in the Los Alamos National 
Laboratory (LANL). It is an object-oriented framework for applications in computational science requiring 
high-performance parallel computers [ 1 ]. This description is based upon POOMA documentation [1,2], 
teleconferences with POOMA developers, and the assessment of POOMA by the National Energy Research 
Scientific Computing Center (NERSC) [3|. 

2.2.2 Synopsis 

Prominent information about POOMA includes the following: 

1 Community of origin— POOMA originates in the LANL. 

2. Description — POOMA is a library of C++ classes designed to represent common abstractions in 
high-performance computing applications. 

3. Team— The POC is Julian Cummings, who is now at the California Institute of Technology. Most of 
the POOMA developers have recently left. 

4. Maturity — The POOMA project started around 1994. (POOMA 2.x is currently under 
development.) POOMA has achieved one of its requirements (i.e., code portability across serial, 
distributed, and parallel architectures with no change to source code). 

5. Users — Most users are within the LANL. 

6. Language — The tools in POOMA are implemented as a C++ class library. Application development 
in C++ using POOMA can be done efficiently by using and/or deriving from those classes with the 
C++ features of inheritance and polymorphism. However, legacy codes in Fortran cannot be 
supported easily in POOMA without major code changes. 

7. Tools/utilities included — POOMA tools and utilities support data types such as field (i.e., array) 
and particle. With those basic data types, application codes can use the tools and utilities such as 
meshes and differential operators. 

• Fields — Fields arc multidimensional arrays representing grids with definable centering. They 
may be fully contained within a processing node, or spread across nodes, according to user 
directives. POOMA predefines field types for the most common element types: scalars (double, 
integer, etc.), vectors, tensors, and symmetric tensors. POOMA users may specify other arbitrary 
element types, although this is not completely straightforward. 

• Particles — An instance of the particle class actually stands for a set of particles, w ith user- 
definable characteristics. Particle sets may also be distributed across nodes, and operations on 
them are expressed in a data-parallei style. 

8. Application interface — Based on utilities provided by POOMA, application codes in C++ can be 
developed quickly through the features of C++ inheritance and polymorphism. 

9. Documentation— Most information on POOMA can be found at [1 ]. 
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10. Associated software — Currently POOMA uses the software package “Cheetah” to wrap Message 
Passing Interface (MPl) and/or Shared Memory (SHMEM) for message-passing operations and uses 
the software package “SMART” to perform multithreaded operations. Multithreaded POOMA 2.x 
programs can he profiled using the Tuning and Analysis Utilities (TAU) library. TAU is a set ol tools 
that allows developers to analyze the performance of distributed and multithreaded programs. It is 
especially useful in tracking C++ classes that arc created and destroyed dynamically. "Cheetah” and 
“SMART” were developed at LANL. 

1 1 . Special features — True portability across serial, parallel, and distributed architectures. 

2.2.3. Description 

POOMA is used to write high-performance Partial Differential Equations (PDE) solvers using finite- 
difference methods on structured, unstructured, or adaptive grids, particle methods, or hybrid methods which 
combine the above. An earlier version of POOMA, generally referred to as POOMA R 1 , has enjoyed 
considerable success meeting these goals in real applications, which include gyrokinclic particlc-in-cell 
plasma simulation, multimalerial compressible hydrodynamics, accelerator modeling, and Monte Carlo 
transport. POOMA 2.x is the next generation of the POOMA software, designed to take advantage of 
advances in C++ compiler technology, multithreaded operation, and a new, highly extensible design. 

As indicated at the POOMA Web site, “POOMA has a flexible array class that supports a plug-in 'Engine' 
architecture to achieve representation independence. It includes a powerful system tor specifying and 
combining domains to construct views of arrays. These views are arrays, so they can be used anywhere an 
array is expected. Using a novel ExprcssionEngine abstraction, array expressions in POOMA arc also first- 
class arrays. POOMA supports multithreaded execution on shared-memory multiprocessors using the 
SMARTS runtime system. An experimental asynchronous scheduler is available that uses data-flow analysis 
to perform out-of-order execution in order to improve cache coherency. POOMA hides the details of parallel 
computation in a flexible Evaluator architecture. For the user, this means that a program can be written in a 
highly abstract data-parallel form, tested and debugged in serial mode, and then run in parallel with very little 
effort. With POOMA version 2.1, physics abstractions like fields, coordinate systems, meshes, efficient 
differential operators, and particles have been introduced." [ 1 ] 

POOMA programs are written at a high level, using data-parallel array expression in the style of High 
Performance Fortran (HPF) or at a serial level using iterators on each CPU. They can achieve high 
performance (comparable to Fortran) thanks to a clever compilation technique called expression templates 
|4). Moreover, POOMA programs achieve true portability across serial, parallel, and distributed 
architectures. 

One of the principal motivations behind POOMA is to provide C++ classes that directly address numerical 
science problems using the methodology of numerical scientists. By managing boundary conditions, and 
supporting efficient evaluation of differential operators, these classes provide the functionality that modern 
numerical algorithms require, and allow' numerical scientists to concentrate on what they want to calculate, 
rather than on how it is to be calculated. 

The main goals of the POOMA framework include the following: 

• Code portability across serial, distributed, and parallel architectures with no change to source code; 

• Development of reusable, cross-problem-domain components to enable rapid application 
development; 

• Code efficiency for kernels and components relevant to scientific simulation; 
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• Framework design and development driven by applications from a diverse set of scientific problem 
domains; 

• Shorter time from problem inception to a functional parallel simulation. 

POOMA is an object-oriented design framework with a layered structure. Figure I below shows the basic- 
structure of the POOMA framework. Application programmers can use and/or derive from these C++ classes, 
which present a data-parallel programming interlace at the highest abstraction layer. Lower implementation 
layers encapsulate distribution and communication of data among processors. 


NPACI Parallel Computing Institute August 1 /-2t, 1998 


The POOMA Framework 


Application 


Algorithm 


Global 


Parallel 


Local 


The POOMA Framework Julian C. Cummings, LANL 


NPACI: National Partnership for Advanced Computational Infrastructure 


Figure 1. The POOMA Framework Design [1] 

As mentioned above, POOMA has used expression templates to optimize performance. By using expression 
templates, it has attempted to solve the problem of poor performance that is inherent in the C++ language. 
The following example shows one of problem origins: 


class Matrix { /*...*/ ) ; 

Matrix A, B, C, D; 

/* */ 

A = B + C + D; 

Suppose that class matrix overloads the operators “+” and “=” to perform element-wise addition and 
assignment. The Final line will be evaluated in a series of binary operations. These will involve temporary 
matrix objects that store intermediate results: tmpl = B + C; tmp2 = tmpl +D;A = tmp2. Creating and 
destroying temporary objects can severely degrade performance, especially if each object contains a 
significant amount of data. This problem has been recognized for some time, and various attempts have been 
made to solve it. The best solution to date is expression templates [4], a flexible and general device that 
avoids the creation of temporary objects. POOMA relies heavily on expression templates to optimize dala- 
paralle! expressions involving particles and fields. POOMA applications thereby retain the benefits of 
overloaded operators with no loss in performance. Since the ROSE preprocessor used in Overture is not 
ready tor public release, it is difficult to compare the technique of expression template with the ROSE 
preprocessor at this time. 
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Thus far, there arc a limited number of scientific demonstrations of POOMA inside the LANL and in some 
universities. Those applications are mullimatcriai hydrodynamics, part icle-in-ccll plasma simulation, Monte 
Carlo neutronies, accelerator physics, volume traction and programmed bum, and numerical tokamak. All 
these applications have been developed in close collaboration with the POOMA development team. 


2.2.4 Evaluation 
Strengths 

POOMA application programs arc written at a high level, using data-parallel array expressions in HPF style 
or at a serial level using iterators on each CPU. Those programs can achieve high performance (comparable 
to Fortran) by using a compilation technique called expression templates. Moreover, they achieve true 
portability across serial, parallel and distributed architectures. The syntax is similar to that of Fortran 90: a 
single assignment fills an entire array with a scalar value, subscripts express ranges as well as single points, 
etc. Since the expression template is a feature of C++, the technique of expression templates can be 
supported by a standard C++ compiler. 

Weaknesses 

The learning curve for POOMA may be steep because ot its extensive use of C++ template leatures. POOMA 
intends to be used for writing a new code in C++ not for adopting a legacy code written in Fortran. The 
biggest problem with POOMA is that the compiling time can be extraordinarily long (hours). Most compilers 
produce long and cryptic error messages if they encounter an error while expanding template functions and 
classes, particularly if those functions and classes are nested. As POOMA uses templates extensively, it is not 
uncommon for a single error to result in several pages ol error messages from a compiler. Finally, some 
debuggers still provide only limited support for inspecting template functions and classes. All of these 
problems are actively being addressed by vendors, primarily in response to the growing popularity ol the 
Standard Template Library (STL). 

Summary 

The POOMA framework has successfully achieved portability across serial, parallel, and distributed 
architectures. However, the extraordinarily long compiling time due to the use ot expression templates 
technique makes POOMA difficult to use for practical applications. But its object-oriented design and the 
methodology used in achieving portability across platforms can be useful for developing an ESMF. 


2.2.5. References 

1 URL: http://www.acl.lanl.gov/pooma/ 

2 URL: http://www.phvsics.ucla.edu/icnsp/Html/mark.htm 

3 URL: http://acts.nersc.gov/pooma/main.html 

4 Veldhuizen, Todd, 1995. “Expression Templates/" C++ Report 7:5 (June, 1995), 26-3 1 . Reprinted in C++ 
Gems, Stanley B. Lippman, ed., 1996. Sigs Books, NY. 


Earth Modeling System Software Framework Survey 


7 



2.3 Cactus 


2.3.1 Introduction 

Cactus 1 1 ] is a software framework resulting from the collaboration of numerous institutions including the 
Albert Einstein Institute (AEI) [2|, Washington University Gravity (WUGRAV) group [3|, National Center 
(or Supercomputing Applications (NCSA) [4|, Konrad-Zuse-Zentruni fucr Informationstechnik Berlin [5], 
International Numerical Relativity Group (TNRG) |6|, Rcchcn/entrum Garching (RZG) der Max-Planck- 
GeselJsehaft [7|, Argonne National Laboratory (ANL) (8|, and Universitat dc les Illcs Balears (UIB) (9|. 

As described on the Cactus Web page, 

"Cactus is an open source problem-solving environment designed for scientists and 
engineers. Its modular structure easily enables parallel computation across different 
architectures and collaborative code development between different groups. Cactus 
originated in the academic research community, where it was developed and used over many 
years by a large international collaboration of physicists and computational scientists. 

"The name Cactus comes from the design of a central core (or flesh) which connects to 
application modules (or thorns) through an extensible interface. Thorns can implement 
custom-developed scientific or engineering applications, such as computational fluid 
dynamics. Other thorns from a standard computational toolkit provide a range of 
computational capabilities, such as parallel input/output (I/O), data distribution, or 
checkpointing. 

"Cactus runs on many architectures. Applications, developed on standard workstations or 
laptops, can be seamlessly run on clusters or supercomputers. Cactus provides easy access to 
many cutting-edge soltware technologies being developed in the academic research 
community, including the Globus Mctacomputing Toolkit, HDF5 parallel file I/O, the PETSc 
scientific library, adaptive mesh refinement, Web interfaces, and advanced visualization 
tools ” [ 1 1 

2.3.2 Synopsis 

Prominent information about Cactus includes the following: 

Community of origin — The Cactus framework originates in the relativistic physics community. 

1 . Description — Cactus is a component-oriented application development environment for scientific 
computing which employs a common interface to components written in multiple languages. 

2. Team — The Cactus team includes at least 25 individuals, a partial list of which includes Gabrielle 
Allen ( 101, Tom Goodafe 1 1 1], Ed Seidel [ 12], Thomas Radke [131, Gerd Lanfermann (14], and 
others [ 15]. The team is actively functioning and is anxious to work with users from other 
communities to further enhance the code infrastructure. 

3. Maturity — As of August 2000, the Cactus team is shipping version 4.0, beta 8.0. Version 1 .0 was 
released on April 24, 1997. Cactus spans at least 3 years and four major releases. The predecessors 
of Cactus go back to the early 1990’s. Cactus benefitted from the NSF Black Hole Grand Challenge 
project and several other attempts to develop a framework for research groups. Cactus has a long 
heritage in community framework development. 
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4. Users — Cactus has approximately 100 users, mostly within the relativistic physics community, but 

recently expanding into other disciplines, such as aerodynamics, condensed matter, chemical 
engineering, geophysics, and climate modeling. Some ol the users, as noted on the Cactus home 
page, are listed below: 

• Within the relativistic physics community, application groups actively using Cactus include the 

following: 

• The AEI 1 2 1 has developed a Cactus Relativity Toolkit. 

• The WUGRAV [3] has developed several Cactus modules. 

• University of Pittsburgh 

• The Pennsylvania Stale University numerical relativity group is using Cactus for its Agave 
[ 16] code in performing 3D evolutions of spacetime. 

• University of Texas 

• University of Timisoara (Romania ) 

• University of Wien 

• University of Washington, Seattle 

• Several groups in Italy 

• Physical Research Lab and Raman Research Institute (India) 

• University of California, Santa Barbara 

• University of Southampton 

• University of Portsmouth 

• Astrophysics users include the following: 

• The cosmology group at NCSA [4], including Michael Norman 1 17|, are porting the 
successful Zeus code to Cactus. Zeus has several hundred users worldwide. 

• The European Union Network Project 1 18], a project simulating collisions of neutron stars 
and black holes, which is closely related to the NASA Neutron Star Grand Challenge project 
[19]. 

• Max-Plank-Institut for Astrophysics 

• The Astrophysics Simulation Collaborator (ASC) [20] is a project sponsored by National 
Science Foundation (NSF) grant PHY 99-79985 to Rutgers University, University of 
Chicago, University of Illinois, and Washington University, that enables large-scale 
simulations in astrophysics. The code is based on Cactus. A progress report [21] provides 
insight into how Cactus can be modified to support new simulation applications. 

• Aerospace users include the following: 

• Deutsche Luft und Raumfahrtzentrum [22], the German Aerospace Center, is working with 
an industrial partner to plug an aerospace application into Cactus. 

• Condensed matter physics users include James Sethna [23j and his condensed matter 
physics group at Cornell University. 

• Chemical engineering groups include Ken Bishop [24] and his chemical engineering group 
at the University of Kansas. 
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Geophysics groups using Cactus include Bosl [25] and the geophysics group at Stanford 
University arc planning to use Cactus. Bosl is also associated with the Digital Earth project 
(26), a project to develop object-oriented coupled models for geophysical modeling. 


• Computational science groups that arc actively using and/or developing Cactus include 

• NERSC [27 1 at Lawrence Berkley Laboratories (LBL) 

• NCSA [4 J 

• The University of Chicago 

• Clcmson University 

• GMD-First in Berlin is working to incorporate Janus, an unstructured mesh code, as a driver 
layer. 

• Mike Holst’s [28) group at University of California, San Diego (UCSD) 

• Charlie Crabb’s [29] group at Lawrence Livermore National Laboratory (LLNL) [30| 

• Imperial College, London, plans to develop an expert system to aid in connecting thorns to 
build an appropriate application. 

• Most recently, Cactus was ported [3 1 1 to the NT Cluster at the Cornell Theory Center (CTC) 
132]. 


• Cactus, having a long history of grid computing experiments, is becoming more extensively 

used in grid communities such as the following: 

• European Grid Project (EGRID) [33 1 

• U.S. Grid Forum |34| 

• The Grid Application Development Software Project (GrADS) [35], a project sponsored by 
the NSF Next Generation Software programs to simplify distributed heterogeneous 
computing, directed by Ken Kennedy [36], uses Cactus as a test grid application. 

• Globus [37] developers 

• The German Gigabit Testbed Project (TIKSL) [38] uses Cactus for their simulation 
environment. 

L Language — The core ot Cactus is written in ANSI C, with PERL heavily used during the 

configuration process. There is no Fortran or C++ in the core code. Cactus currently accepts thorn 
modules in Fortran 77, Fortran 90, C, and C++. Support for other languages such as Java. Practical 
Extraction and Report Language (PERL), and Python is planned in the next version release (4.1). 

2. Tools/utilities included — Cactus comes with several toolkits, mostly oriented toward relativistic 
physics. It also has a runtime system with a scheduler. 

3. Application interface — Cactus employs a custom common interface to thorns (component 
modules) which is language independent and provides some object-oriented capabilities. 

4. Documentation — The Cactus team actively maintains a Web site [ 1 ], which provides links to a host 
of information. Printed documentation includes a user’s guide [39], quick start guide, and a set of 
developer tutorials [40], [41 ]. Additionally, the Web site provides access to numerous presentations 
[42], (43 1, (44 L (45], [46], [47], L48], [49], [50], [511, [521, [531, [54], publications about Cactus 
[55], [56], [57], [58], [59], and application publications and posters [60], [61], [62|, [63], (64], [65], 
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1 66 1, [67], [68]. Cactus has also been the focus of a workshop [69], [70) held at NCSA [4 1 . Thus, 
Cactus has a significant amount of associated documentation in its user community. 

5. Associated software — Cactus requires PERL [71 1, Gnu Make [72] and a C/C++ compiler. It also 
interfaces with the following software: 

• Concurrent Version Systems (CVS) [73] — Source code configuration management 

• Cygwin [ 74 1 — Unix utilities for Windows platforms 

• MPI and MP1CH [75] — Message-passing interfaces for parallelism 

• Globus [76] — Toolkits for computational grids 

• FlexIO [77 1 — Application program interface (API) for storing multidimensional seienlilic data 

• HDF5 [78] 

— File format for storing scientific data 

• IEEEIO [79 1 — library for storing multidimensional data in binary format 

• LCAVsion [80] — scientific visualization tool 

• Amira[81] 

— Scientific visualization tool 

• Portable Extensible Toolkit for Scientific Computing (PETSC) [82] — suite ot data structures and 
routines for scalable (parallel) solution of PDE problems. 

• Grid Adaptive Computational Engine (GrACE) [83] 

• Autopilot [84] 

— Real-time adaptive resource control 

• OpenDX [85] — visualization 

• Performance Data Standard and API (PAPI) [86] — performance monitoring library 

• Panda [87] 

— Parallel I/O library 

1 . Performance — So long as most of the execution time is spent inside a module/thom, performance 
of a climate application using Cactus should be about the same as belore. The Cactus scheduler does 
not play a large role in the execution time. Cactus achieved 142 Gllops on a 1024 Cray T3E-1200 
with a full thorn set solving the coupled Einstein-Hydro equations. 

2. Special Features: 

• Unique component interface spans Fortran and C and supplies inheritance capability tor data 
declared using the interface. 

• Cactus applications can include component modules that allow them to function as Web servers 
[88]. The running application can be inspected via a Web browser through which the user can 
see the configuration of the application, the currently running modules, the values of the 
parameters, and graphics showing intermediate results. Application parameters can be 
configured as steerable, enabling them to be changed via the Web browser during the application 
run. The Cactus homepage 1 1 ] provides a link to a perpetual Cactus run. 
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• Cactus is especially strong in remote visualization, steering, and grid computing and is used 
heav ily in development of these new technologies worldwide. 

• The Web server now allows fine control ol the simulation, such as pausing, terminating, and 
checkpointing. 

• The Web server interlace is being currently developed and will increase in features and 
functionality. 

• Applications developed with Cactus arc automatically grid-enabled and can make use of 
advances in grid computing. 

• There arc several projects and modules for remote access to visualization and control. 

• An advanced portal for Cactus is being developed at ANL, in collaboration with researchers 
at LBL, NCSA, Washington University, and AEI, to facilitate all aspects of Cactus 
configuration, parameter input, job management, remote submission on resources, and job 
monitoring and steering. This capability will be demonstrated at the Supcrcomputing 2000 
Conference. 

• With respect to rewriting codes to use Cactus, there arc several levels of integration. On the 
low end, there is little work and virtually no code intrusion. An entire code may be plugged 
in but may only use parallelism. At the high end, with more work, an existing code makes 
intense use of the Cactus architecture. In this respect, the rewrite effort scales the same way. 
The code can be shifted from the low end to the high end, as the programmer wishes. 

2.3.3 Description 
High-Level Design 

Because ot the high complexity o! solving relativistic equations [58], described as “among the most 
complicated seen in mathematical physics,” members of the relativistic physics community created Cactus as 
a common development tool which allowed multiple components, known as thorns, to be assembled and run 
in the same environment. The need for a modular system is driven by the desire to have members of the 
community work relatively independently on various software components. The need for multiple language 
support is derived from the lact that community members arc developing components in various languages, 
including Fortran 77, Fortran 90, and C. Thus, these needs, in conjunction with other needs, translate into a 
need for a single development environment with a common interface that can span multiple language 
modules. The Cactus high-level design is governed by the following major ideas: 

1 . “A framework is a reusable semi-complete application that can be specialized to produce 
custom applications” (89| 

Cactus is not just a toolkit, a library, or a module that performs a particular function. It is a complete 
framework that meets the definition criteria, oriented towards the production of complete 
applications consisting of interacting objects. 

2. Fortran is a crucial scientific language that must be supported because speed is important. 

John Backus, a Fortran designer, was quoted in The Grid [90], [91 ] (p. 184) as saying the following 
w ith respect to the Fortran language: 

“It was our belief that if Fortran, during its first months, were to translate any reasonable 
4 scientific' source program into an object program only half as fast as its hand-coded 
counterpart, then acceptance of our system would be in serious danger... To this day I 
believe that our emphasis on object program efficiency rather than on language design was 
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basically correct. I believe that had we failed lo produce efficient programs, the widespread 
use of languages like Fortran would have been seriously delayed.” [92] 

For similar reasons, rather than abandoning Fortran and choosing an object-oriented language that is 
much more suitable lo the design ol their system, the Cactus team has pul a great deal ol cllort into 
accommodating existing Fortran code. 

3. Modules should be self-contained with small interfaces where all communication takes place 
through the argument list. 

James Hack [931, in a discussion on modularization and coupling of climate models said the 
following: 

“Historically, scientific progress in atmospheric modeling has been slowed by technical 
difficulties of incorporating and testing physical parameterizations in dillercnt large-scale 
numerical models. Such problems arc fundamentally linked to the fact that most codes are 
not modular in their design and often make use of very different data structures. 

Recognizing this problem, a number of scientists from major atmospheric modeling 
institutions have adopted a set of coding rules to make physics packages more plug- 
compatible in order to facilitate their easy exchange [94]. Although this set ot rules is 
specifically intended for physics packages that do not have a large number ol their ow n 
prognostic variables, the conceptual approach is appropriate, and will ultimately prove 
necessary, for coupling complex climate system components (ocean circulation models, 
chemical models, biosphere models, etc.). Some of the more important concepts contained 
in this coding standard are 

• Each component should refer only to its own subprograms and a limited set of 
standard intrinsic functions. 

• Each component should provide for its own initialization ol static intormation and 
initial data through a single initialization entry point. 

• All communication between packages shall take place though the argument list 
associated with a single unique entry point into each package. 

This approach to coupling major model components is quite reasonable.” ([95], p. 317). 

Cactus works best with modules written in this style, where all communication with a 
module takes place through a single entry point using only parameters in the argument list. 

Modules using other means of communication or with multiple entry points will have more 
difficulty working with the Cactus system. 

Cactus effectively integrates these three ideas at a high level. 


Use of the System 

The enabling technology in Cactus is associated w ith the need for language-independent parameter and data 
passing. In Cactus, this is accomplished by having the user 

1. Create special files in the module directory, independent from the source code, to declare and 

specify dominant parameters and data. These files provided a means to create data “objects” using 
inheritance and with public, private, and protected variables. Thorns may be scheduled in groups, 
with other routines/groups being scheduled before or after. A “schedule while (condition)” construct 
is also allowed to allow' for some dynamic control and for loops. This is more primitive than a 
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scripting language, except for dynamic change at runtime, hut a scripting language is planned for the 
4.1 release. 

2. Insert special Cactus keywords in the source code subroutine calling parameter lists instead of 
variable names. 

3. Compile the application using the Cactus system. 

Cactus, in turn, builds the application by 

1. scanning the special files to learn about the variables; 

2. inserting variable names automatically in the calling lists for each subroutine; 

3. compiling the final application using standard compilers. 

When the application runs. Cactus performs the following functions: 

1. loads thorn s/modules; 

2. schedules functions from thorns (note that there is more than one entry point to a thorn. They arc not 
treated as monoliths.); 

3. transfers data between thorns; 

4. provides interfaces to visualization tools; 

5. provides interfaces to external Web browsers; 

6. writes data. 

The enabling technology, then, is the Cactus source code, PERL [71 ) scripts, and interface conventions which 
enable it to accomplish these tasks. PERL is a language that is traditionally used for text processing because 
of its powerful pattern recognition capabilities. With Cactus, PERL and the C- preprocessor (CPP) are used as 
part of the build process but are not part of the runtime. Cactus provides easy parallelization for modules. 


Views of the System 

Cactus is a comprehensive system that can be described from several perspectives, each providing insight 
into what Cactus can do: 

• As a language extension to C and Fortran. 

• As an object-oriented language. 

• As a compiler. 

• As a development environment. 

• Asa runtime environment. 

• Asa language extension. 

As Cactus requires the user to insert special keywords in the source code, Cactus is a type of language 
extension. From this perspective, Cactus is similar in approach to High Performance Fortran ([96] and [91 ], 
p. 188), an extension to Fortran in which areas of data parallelism can be denoted in the code via comments 
or several new keywords. Cactus is different from HPF in that the new keywords do not denote regions of 
parallelism but instead denote regions of the code where Cactus needs to substitute variable names at compile 
time. In a similar vein, whereas the principal focus of NPF is the layout of arrays ([91], p. 188), one of the 
main focuses of Cactus is storing array information in objects through an inheritance mechanism. However, 
Cactus is not an extension to a single language (such as Fortran), but is instead an extension to multiple 


14 


Earth Modeling System Software Framework Survey 



languages (Fortran and C) using the same keywords. Because multiple languages are involved. Cactus also 
specifics unique names for types such as CCTK^REAL which corresponds to a real in Fortran or a double in C. 


An Object-Oriented Language 

As it provides the means for users to build data structures in terms of other data structures using an 
inheritance mechanism, Cactus is a type of object-oriented language. Inheritance is a powerful feature of 
object-oriented languages and a key part of object-oriented frameworks ([89|, p. 5). Inheritance is missing 
from both Fortran and C. Cactus currently provides this inheritance for data structures but not for method/ 
function names, thus allowing “subclasses" to be derived from previously defined data structures. Inheritance 
for function names will be available in the final release via function aliasing. 

In addition to having inheritance. Cactus is like object-oriented languages in that thorns are similar to classes 
both in the manner of specification and in operation. For example, for a hypothetical MyClass class definition 
in the C++ language, the class variables are traditionally specified in two files: (1) MyClass. h, containing 
declarations, and (2) MyClass .C, containing member functions. Each member function of MyClass 
automatically has access to all data elements declared as part of the class in MyClass.h without passing in the 
function argument list. If the programmer decides that MyClass requires an additional data member, such as 
a gridded three-dimensional array, Z, then Z is added to MyClass.h and becomes automatically available to 
all member functions of MyClass. Thus the language provides the means to “pass" class variables to member 
functions without changing the function prototypes. In a similar fashion Cactus provides a means to declare 
“member data" in files separate from the thorn source code and automatically “pass" these to the thorn 
functions without requiring the user to change the function prototype. 

Though Cactus is similar to object-oriented languages in form and function it is unlike them in elegance. 
Cactus is awkward in both form and function. Instead of a single declaration file (MyClass.h), three are 
required ( interface.ccl , param.ccl, and schedule.ccl). Instead of meaningful type names like real or double, 
awkward names such as CCTK_REAL are used for Cactus-owned variables. Some of this awkwardness is 
unavoidable, a direct byproduct of trying to make something that interoperates between two dissimilar 
products. In the same way that a stereo system designed to operate in either a bus or a motorcycle, w ith two 
different mounting environments and two different acoustical environments would be awkward, the Cactus 
design is awkward because it straddles two different languages, thus requiring awkward conventions. 
Nevertheless, even the awkwardness can be a benefit in some cases, as CCTK_REAL can be defined as 
double or float, enabling users to easily configure the degree of precision in a computation. 

A Compiler 

In the sense that Cactus scans source code and looks for keywords, it is a type of compiler. Cactus uses GNU 
make and PERL scripts to scan the source code, perform substitutions and arrange for the overall build of the 
application. It also relies on the CPP to perform macro substitutions. Because it calls the other compilers, the 
CPP is a type of super-compiler that spans the multiple languages. 

A Development Environment 

As it manages the production of the final application, Cactus is a type of development environment. The user 
specifies code in thorns and Cactus provides assistance in assembling the components. 
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A Runtime Environment 


As it provides scheduling ol the various modules during program execution. Cactus is a type of runtime 
environment. Cactus does not just compile source code written by somebody else, it also inserts specific 
code ol its own to provide runtime scheduling between the different thorns. At a high level this is similar to 
the Parallel Application Workspace (PAWS) environment [97J where a central controller coordinates the 
running ol dillerenl components. At a low level this is similar to a language like Java which generally runs on 
a virtual machine which creates, schedules, and controls the objects. This does not mean that there are two 
levels ol control. It means that Cactus can be viewed in these two ways. 

2.3.4 Evaluation 

Cactus is unique among the frameworks we examined because ol its comprehensive modeling approach. It is 
worthy ol close consideration by the climate community, not only for its implementation, but more 
importantly for its method. In many respects, the needs of the climate community are not significantly 
dillercnt than the needs of the relativistic physics community or of any other community that develops and 
uses complex software and computer systems. The comprehensive manner in which Cactus addresses these 
universal needs provides valuable insights into what a framework should be and can become. 

Cactus addresses the issue of how to promote the development of interchangeable software within a 
community by providing a tool with a comprehensive focus. The genius of the Cactus framework is not so 
much in the individual software elements as in the comprehensive nature of the solution, which enables the 
business of assembling and running an application to be effectively conducted. Individually, the elements 
may be either highly desirable to the climate community or, alternately, distasteful and awkward. But even 
some of the awkward elements, as part of a larger system, have a strength that other frameworks do not 
provide. 

Strengths 

These arc the issues that the Cactus framework addresses: 

1 . Cactus is a Iramew ork in the true sense. It addresses the need for a community software focal point 
by defining the nature of the soltware components and the relationships ([89] p.4) between them and 
by providing a common tool to assist with assembly and scheduling. 

2. According to the pattern of true frameworks, Cactus provides a mechanism for inversion of control 
([89] p.5), a supervisory process (flesh) which schedules the interacting objects (thorns), thus 
relieving the module developer of that responsibility. 

3. Cactus addresses the reality of researchers working independently by developing the thorn/ 
arrangement concept ([39], Chapter B1 ) which spans code, interface, and documentation. These 
concepts provide a common lormat to use that makes code more interchangeable and understandable 
among members of the community. 

4. Cactus addresses the reality of researchers working on a range of platforms because it is relatively 
platform independent, running on platforms ranging from laptops to supercomputers. 

5. Cactus simultaneously addresses two of the most difficult realities facing scientific computing 
today: 

• There is a large body of working, debugged, efficient legacy Fortran code with which 
researchers arc intimately familiar. 
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• The majority of training and new developments in eomputer science are taking place in newer 
languages such as C, or object-oriented languages such as C++. 

Cactus addresses these issues by defining a single interface that can be used with either language. 
The interface includes definitions of data type and array structures, and provides rudimentary 
inheritance capabilities that can be used with Fortran 77, Fortran 90, or C code. Thus, Cactus 
provides access to some object-oriented features without forcing users to migrate all of their code to 
an object-oriented language. This is a much better solution than dclining an interlace at the lowest 
level, corresponding to the intersection of languages. 

6. Cactus addresses the need for researchers to inspect a running process via the Web [88]. Cactus 
provides utilities that allow Web browsers to connect to a running process for inspection. The Cactus 
home page [ 1 1 provides a sample connection. It is important to note that it is not only possible to 
inspect via a browser, but that it is also possible to retrieve any data from the memory of a running 
simulation and have it streamed in HDF5 format to a local host, for more in-depth visualization and 
analysis. 

7. Cactus has a strong role in the emerging area of Grid computing because it enables applications, 
which are not Grid aware, to run in a Grid environment, thus saving the researchers the effort of 
building in this ability. 

Weaknesses 

Cactus has the following weaknesses: 

1 . The standard Cactus arrangements |98) include little code that is directly applicable to the climate 
community because they are oriented towards Cartesian grids instead of geographical grids. Current 
arrangements include 

• Base — boundary conditions, Cartesian coordinates, symmetry conditions, general I/O, scalar 
and screen output, time step 

• Elliptic — solvers for elliptic equations 

• PUGH — parallel driver layer based on MPI, three-dimensional parallel interpolator 

• PUGHIO 

• Cactus WaveToy — Wave Evolver used as a demo 

• Net 

• External 

Therefore, the climate community would have to devclop/adapt their own code and modules. 
Nevertheless, as demonstrated by the spectrum of users, the Cactus team is anxious to work with 
communities like the climate community to extend Cactus functionality in specific ways. 

2. The scheduler may not be as flexible as desired for some climate simulations. Some climate 
simulations run models sequentially with different numbers of iterations for each model with 
intermediate coupling exchanges. This can be accommodated to some degree using the WHILE 
functionality in Cactus. Nevertheless, having a real scripting language would provide more powerful 
capabilities. 

3. The common interface between Fortran/C is awkward. By moving the subroutine variables out of 
the subroutine into separate files it becomes harder to see what is being passed in and out. 
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Nevertheless, hidden arguments arc only those associated with the scheduler. All other arguments 
can he declared in the code as before. 

4. Saying that Cactus works with Fortran, C, etc., is slightly misleading. Once a code is converted to 
Cactus style it is committed and can he used on no other platform. If the Cactus-modified interface 
subroutine is small, then all ol the lower-level subroutines can be pure Fortran or C so this isn't a 
problem. At the same lime, however, this limits the benefits ol data “objects” because they have to 
be immediately decomposed into variables and passed around. On the other hand, if the choice is 
made to pass around data as Cactus “objects’ at all levels, then the code is fully Cactus-izcd and will 
not compile or work on any other platform. Thus, when a community chooses Cactus, it must do so 
with a lull commitment. This is a weakness only in cases where interchanging code with non-Cactus 
communities is important or where Cactus cannot be a complete solution for the climate community. 
Where such an exchange is necessary, one way to get around this problem is to define a macro to 
refer to CCTK_ARGUMENTS when using Cactus and to refer to the actual list of parameters when 
not using Cactus. 

5. Cactus does not have a scripting language, as does ROOT |99], a capability that could be helpful. 
This capability is planned for release 4.1 

6. The type of inheritance supplied by Cactus is not the same type of inheritance that is integrated into 
object-oriented programming languages. Therefore, it has a limited ability to solve the types of 
problems that inheritance is typically used for. 

Summary 

Cactus is a complete framework solution to scientific modeling. It addresses the issue of differences between 
Fortran and C with a common interface and provides a development environment and a runtime environment, 
including a scheduler, lor the modules. It also provides a means to inspect running code. It has a large 
amount ol documentation, is used by many organizations and is stable, and has a supportive development 
team. The main disadvantage with Cactus is that it may require some adaptation before it can be used by the 
climate community. Nevertheless, the Cactus team is highly motivated to work with the climate community 
and other communities to extend Cactus capabilities. 
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2.4 Overture 


2.4.1 Introduction 

Overture [ 1 ), [2| originates in the Lawrence Livermore National Laboratory (LLNL) and the Los Alamos 
National Laboratory (LANL). The project is now consolidated into the LANL. Overture is an object-oriented 
software system for solving Partial Differential Equations (PDE) in serial and parallel environments. It 
consists of a library of C++ classes for writing serial and parallel PDE solvers using overlapping and/or 
adaptive block-structured grids. Overture programs are written at a very high-level, using data-parallel array 
expressions in the style of HPF. It can achieve high perlormance (comparable to Fortran) through a source-to- 
source (C++ to C++) preprocessor called ROSE. Effectively, ROSE is a counterpart to the expression 
template technique of the POOMA framework [3]. This survey is based upon Overture documentation, 
teleconferences with Overture developers, and the assessment of NERSC on Overture [4). 


2.4.2 Synopsis 

Prominent information about Overture includes the iollowing: 

1 . Community of origin — Overture originates in the LLNL and the LANL. Now the project is 
consolidated at the LLNL. . 

2. Description — Overture provides a portable, flexible software development environment lor 
applications that involve the simulation of physical processes in complex moving geometry such as 
modeling the motion of a submarine. 

3. Team — Present members of the Overture team include David Brown, Bill Henshaw-, and Daniel 
Quinlan. 

4. Maturity— This project began in the early 1990s. The parallel version of Overture is still being 
developed. All serial implementations require OpenGL (a graphics library). Overture is based on a 
few other software products, among which are the A++/P++ array library and the ROSE 
preprocessor. Development of these products is not yet complete. The main issues awaiting 
resolution are 

• The ROSE preprocessor (needed to ensure adequate performance of aggregate array operations) 
can currently optimize with hints; eventually it will optimize without hints. 

• Overture runs in a parallel environment, but only for single-grid applications because of some 
missing features in P++. 

• P++ has only been ported to a network of workstations (Ultra Spares ), although it should be 
quite easy to port to most common parallel architectures 

5 User — most users arc inside the LLNL and the LANL. During the last several years, Overture has 
been used to develop How solvers for high-speed compressible How problems, incompressible How 
problems, low Mach number and non-Newtonian fluid How. One National Oceanic and Atmospheric 
Administration (NOAA) organization at Boulder is trying to use Overture for ocean modeling. 

6. Language — Overture is written mostly in C++ and the A++/P++ array class library. Using the 

serial/parallel array class library A++/P++, efficient and portable serial or parallel code is generated. 
An application program has to be written in C++ if the application program wants to use the features 
of Overture, especially in the area of parallel communication. 
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Tools/utilities included — the Overture C++ classes provide tools (or the rapid development of 
application codes. The main class categories are listed below: 

• A++/P++ arrays describe multidimensional arrays and provide for serial and parallel operations 
on those arrays. In the parallel environment, these provide lor the distribution and interpretation 
ol communication required lor the data parallel execution of operations on the arrays. 

• Mappings define transformations such as curves, surfaces, areas, and volumes. These are used to 
represent the geometry of the computational domain. 

• Grids define a discrete representation of a mapping or mappings. These include single grids and 
collections of grids, in particular composite overlapping grids. The Ogcn (a overlapping grid 
generator for Overture) provides tools for the construction of curvilinear grids, and for 
overlapping those grids to represent complex moving geometries such as submarine. 

• Grid functions provide for the representation and centering of solution values such as density, 
velocity, and pressure, defined at each point on the grid(s). 

• Operators provide discrete representations of differential operators and boundary conditions 
through Unite difference or finite volume approximations. 

• Visualization tools based on OpenGL are provided to furnish a high-level graphics interface for 
visualizing geometry and simulation results. 

• Adaptive mesh refinement provides automatic refinement of the overlapping grid structure for 
increased local resolution and efficiency of computational simulations. 

• Load-balancing tools are provided for automatic load-balancing of computations on the adaptive 
overlapping grid structure on Parallel computers. 

• Parallel distribution mechanisms are provided through the PADRE library, part of the 
Department of Energy (DOE) 2000 ACTS toolkit. 

Application interface — based on the utilities described above, application codes written in C++ can 
be developed quickly by use of the features of C++ inheritance and polymorphism. Since the 
Overture utilities are based on the A++/P++ array class library which is written in C++, an 
application cannot be written in Fortran easily. 

7. Documentation — Overture has a good documentation located at http://www.llnl. gov/casc/Qvcrturc 

8. Associated software — Overture parallelism is implemented using the MPI and PVM message 
passing libraries, PADRE (a library for the description of distributions of data and the generation of 
communication schedules to address their communication requirements), and a HPC++ standard 
threads library (TULIP). In addition OpenGL is used for visualization. 

9. Special features — the developers of Overture create and use the ROSE preprocessor to optimize 
C++ performance. In addition. Overture has aggregate array operations (similar to those in 
POOMA) and tightly integrated graphical features based on OpenGL. AMR++, a package that 
directly supports adaptive mesh refinement methods, is built on top of Overture. In the future, 
Overture will add support for unstructured grids and an improved ROSE preprocessor. 

2.4.3 Description 

The main objective of the Overture project is to provide a flexible code development environment for 
applications using adaptive overlapping grid technology which addresses: 
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• complex overset grid data structures; 

• need to rapidly develop application codes for existing and future application areas; 

• need for software that is portable across multiple architectures irom workstations to massively 
parallel platforms, while maintaining at least Fortran 77 performance. 

Overture is an object-oriented framework with a layered structure. Figure 2 shows a block diagram that 
represents the hierarchy of the class libraries that define the Overture framework. Overture includes about six 
layers. All levels but the application layer arc considered part of the Overture distribution (applications arc 
the property of the application developers in general ). C++ optimizing preprocessor, adaptive mesh 
refinement, turbulence models, and front capturing arc still in development and not a part ol the public 
distribution of Overture. Documentation for each library is available on the Web separately. The lollowing is 
a brief summary of the Overture layers: 

• Communication, data distribution, and threads— This lowest level within the Overture framework 
represents a foundation of parallel libraries upon which the rest ot Overture is built. This layer 
includes the MPI and PVM message passing libraries, PADRE, a library for the description of 
distributions of data and the generation of communication schedules to address their communication 
requirements; and an HPC++ standard threads library (TULIP). 

• Serial/parallel program interface — This layer represents the principal interlace lor the 
development of applications within Overture. As a single layer ol the hierarchy, it represents a 
hierarchy of program interfaces along with abstractions tor the representation ol complex geometry, 
mapping objects; discretizations of the geometric surfaces, mapped grid objects, defined upon those 
mapping objects; and the representation of data on those grids, grid function objects. Separate 
container objects are used to represent the collections ot each of these mapped grid and grid function 
objects, thus forming “mappedgridcol lection” and “gridcollectionfunction” objects. The details ol the 
representation of the grid function objects are partly encapsulated within the A++/P++ array class 
library, which forms a fundamental level and abstraction within Overture. The A++/P++ Array Class 
Library' provides the principal mechanism by which parallelism is encapsulated (hidden, to some 
extent). The array class also is central in addressing the pcrlormance issues lor dillcrent 
architectures; the work on the array class library specific to performance is considerable. The 
purpose of the ROSE C+ + Optimizing Preprocessor is to isolate the mechanisms by which 
performance is addressed in the optimization of the array class objects and how they are used within 
an application and within Overture directly. More details on how performance is addressed within the 
A++/P++ array class can be obtained from the A++/P++ documentation and related papers (If File 1/ 
O for all Overture objects is provided via HDF format file structures. The highest level ol the 
hierarchy within the scrial/parallel program interface layer is represented by the Operator Library'; 
this class library includes both finite volume and finite ditlerence operators (div, grad, curl, laplacian, 
different orders of derivatives, etc.). The advantages ol this level ol abstraction is that it permits the 
development of applications using high-level code remarkably similar to the fundamental 
mathematical equations themselves. Finally, the serial/parallel program interlace layer includes the 
graphics and visualization required for real-time graphics, post-processing graphics, movies, and 
more complex visualization as required for physical simulation using numerical techniques. The 
visualization understands the geometry represented by the Overture objects so that applications 
obtain a complete visualization solution. 

• Numerics — This layer of the Overture framework includes elliptic solver, adaptive mesh refinement, 
and grid generation applications. 

• Computational Fluid Dynamics (CFD) — This layer of the Overture framework includes How 
solvers, some of which arc available within the Overture primer manual and are particularly short 
and simple to express in the high-level mechanisms represented by the scrial/parallel program 
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interface. Separate libraries isolate current work on turbulence models and front capturing (using 
level-set methods). 

• Combustion chemistry (interface to Chemkin) — This layer of the Overture framework represents 
an interface to the Chemkin software available through Sandia National Laboratory. 

Application layer — This top level defines the highest level of interaction between an application and 
the Overture framework. An application may, and typically would, interface through multiple levels 
of the hierarchy represented by Overture. Figure 2 shows a couple of different applications (including 
weather, ocean modeling) being developed by other groups using Overture and interacting with 
Overture through different levels of the hierarchy. The developers of Overture did not provide any 
detailed information on how Overture was used in the climate community. It is not clear from figure 
2, but each application additionally interacts with some level of the serial/paralle! program interface 
as well. All internal data (e.g., geometry) is similarly available to C and Fortran applications. 


Design of the Overture Framework 

Our Principle Application Domain Other Application Domains 



In general, the performance of C++ is not as good as Fortran for numerical computations. To satisfy the 
requirements of the Overture framework, the performance has been addressed using multiple mechanisms, 
both internal and external to the C++ language. Three techniques have been investigated: 


• Binary overloaded operators — Although highly optimized in Overture, the use of binary operators 
results in only half the performance of optimized Fortran in practice (see figure 3) and performance 
drops in half again for stencil operations on cache-based machines because reuse of data in cache is 
poor. However, the method is extremely portable and compiles quickly. 

• Expression templates — Although the performance is generally superior to binary overloaded 
operators this is most clear only on stencil operations where the greatest reuse of cached data is 
possible. But, the expression template technique is only a single statement optimization mechanism 
and as such it is limited in its applicability. This mechanism uses the C++ template mechanism so 
aggressively that it lacks portability and results in compile times that increase astronomically (factors 
of 3,500 times slower) making program development difficult at best. 
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• Optimizing preprocessor (sourcc-to-sourcc transformation} — To avoid the inelliciencics of low- 
level abstractions within framework s/1 ibrarics, preprocessor mechanisms (see figure 4) have been 
developed to permit architecture-specific optimizations not possible within libraries. Applications 
may in the future use mechanisms to permit their optimization using the additional semantics lound 
within the user's use of the framework. The compiler will still not know the semantics of the 
framework, but the introduction of a third process, an optimizing preprocessor, can provide more 
aggressive parameterized transformations of the user's expressions using the framework's more 
restricted semantics. This mechanism uses the Sage II (an object-oriented toolkit lor building 
program transformation systems for Fortran77 including 90, and C, and C++ languages). More 
information is available at the Sage Web site |5J. On cache-based architecture systems the Overture 
framework achieves Fortran 77 performance, at a minimum, and can often achieve two to tour times 
better. This work represents the newest addition to the Overture framework and is the result of 
significant focus on performance as Overture has matured. 

Figure 3 shows typical performance for Fortran 77, C++ (overloaded binary operators), C++ with expression 
templates, and C++ optimized in the manner of the ROSE preprocessor. Part of the research of Overture 
developers has been in the development and exploration of these different techniques. 



Operands 

Figure 3. Comparison of Execution Models 1 1 ) 
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Application 



Strengths 

Through its object-oriented design, Overture reduces code duplication, encourages interoperability of 
application software, and simplifies the learning curve for new computational methods if a user knows C++. 
Overture’s object-oriented architecture provides flexibility to address a wide range of applications that 
involve simulations of complex moving geometries on serial and parallel computers. The advantages of this 
approach include reduced code development time and broader, more in-depth research into numerical 
methods for scientific and industrial applications. 

Weakness 

Since A++/P ++ is the basic data type used in the Overture framework, application codes have to be written 
with A++/P++ array which means that application codes have to use C++. Legacy code written in Fortran 
may not be easily ported and supported in the Overture framework. Currently, Overture does not have grid 
utilities for climate modeling. 

Summary 

The Overture framework has successfully demonstrated that complex computational problems, such as 
solving PDE for overlapping grids on parallel computers, can be effectively solved with the help of an object- 
oriented framework. An application written in C++ can take advantage of the tools and utilities in the 
Overture f ramework, while an application written in Fortran cannot do so easily. Currently, the Overture 
framework docs not have the grid utilities for climate modeling. Therefore, a set of geographical grids would 
have to be developed by deriving from the C++ class library of Overture. 
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2.5 GEMS 


2.5.1 Introduction 

Goddard Earth Modeling System (GEMS) is developed by the NASA Seasonal to Inter-annual Prediction 
Project (NSIPP) and Data Assimilation Office (DAO) organizations at NASA Goddard Space Flight Center. 
GEMS provides Fortran90 tools/utilitics lor developing climate modeling applications. A few years ago, 
DAO and NSIPP worked together to outline the requirements and design for GEMS. Later, DAO and NSIPP 
separately developed their own versions of GEMS. Since the DAO’s version of GEMS is still under 
development, this report addresses the NSIPP’s version of GEMS. This survey is based on information 
contained at DAO’s Web site 1 1 1 and review of the GEMS source code of Max Suarez’s group. 


2.5.2 Synopsis 

Prominent information about GEMS includes the following: 

1. Community of origin — NSIPP 

2. Description — It provides tools/utilities for climate modeling. 

3. Team — Max Suarez and others in NSIPP. 

4. Maturity — It has been used for developing NSIPP’s production code. 

5. Users— NSIPP 

6. Language — GEMS is written mostly in Fortran 90 and contains some Perl. Applications can be 
written in Fortran 90. 

7. Tools/utilities included — The GEMS framework consists of parallel utilities (parallel 
communication and wrappers), grid utilities (grid manipulation), couplers (exchange information 
among application components), clock (time and alarm), array (expand and trim array), and other 
utilities such as conversion of double precision for different computers. In the parallel 
communication utilities, both SHMEM and MPI message passing protocols are wrapped so that the 
codes can be run on various multi- or single-processor computer platforms. (Wrapper here means 
that a new set of instructions is introduced to cover the SHMEM and MPI instructions so that a user 
does not use SHMEM or MPI directly.) 

8. Application interface — A group ot couplers has been developed to couple various application 
components (Ocean, Dynamics, Fast Physics, Slow Physics). 

9. Documentation — Very limited documentation is available (DAO’s office Notes [ 1 J and GEMS’s 
source codes). 

10. Associated software — MPI and SHMEM utilities have been used in parallel communication. 

1 1. Special features — GEMS provides basic tools and utilities required for climate modeling. 
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2.5.3 Description 

GEMS developer’s interpretation of object-oriented design in the case of global Earth science modeling is as 
follows: 

“The main goal of object-oriented design is to modularize data, and data transformations, 
thereby making the system robust to generalizations in the data structures. The object- 
oriented paradigm seems particularly useful for global Earth science modeling, with models 
and observations hav ing their own data structures and complex transformations. In a 
distributed memory environment, a signil leant portion ol the parallel algorithm is associated 
with transformations from one model grid to another, or from model grid to observation 
locations. By encapsulating model data structures and model grid transformations, a great 
deal of code reusability can be realized, at the same time ensuring that changes in one model 
do not unnecessarily propagate throughout the system.” [1] 

Based upon this understanding of object-oriented frameworks, the concept of GEMS framework has been 
developed: “As put forward by Max Suarez, GEMS is made up of a collection of models, each operating on 
its own grid and on its own state, and a collection of coupler routines which convert from one model grid to 
another. The resolution and orientation of each model grid is dictated by the physics and numerics of the 
modeled process, rather than by programming constraints imposed by the rest of the code. For example, the 
calculation of diabatic heating due to long-wave radiation is extremely CPU-intensive and has become 
problematic with the demand for high horizontal resolution within the dynamics. Using a coarser horizontal 
grid within the long-wave radiation model may significantly reduce CPU time while having little impact on 
the system accuracy. Results may in fact improve if the reduced CPU time could allow for more frequent 
radiation calls using more frequent cloud inlormation ” [ 1 1 

Figure 1 provides the high-level design of the GEMS systems. In the figure, Fortran 90 modules are denoted 
with circles. Application modules supported by GEMS consist ot ocean, dynamics, slow physics and last 
physics, and diagnosis (LSM_DIAG and LLS_DIAG) utilities. The GEMS framework provides couplers for 
those application components to exchange information and clock utilities for controlling the running 
sequence. In addition, the GEMS framework also provides the utilities such as grid and parallel 
communication for readily developing application components. 



Figure 5. Relationship Between Application and Framework in Aries/NSIPP 
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The detailed description of the code structure of GEMS can he found in the following: 

“First, GEMS compliancy requires the standardization of model and coupler interfaces, as 
well as model utilities (e.g., initialize, run, finalize), thus allowing greater case in writing 
applications. Memory management within a GEMS model is based on dynamical allocation, 
and controlled by generic New() and DeleteO routines. Interaction between models is 
channeled through the coupler, which translates quantities from one representation into 
another. Fortran 90 data structures are used for definitions of the model states and couplers to 
allow data abstraction and to (acilitate multiple instances of the model system within the 
application. The concept of a class in object-oriented design includes data structures as well 
as the routines (member functions or methods) which operate on the data structures. In the 
GEMS framework there are two main classes: 

1. Model class — As the name suggests, a model class is used to represent the environmental models 
comprising GEMS. A model class has 3 main data structures: 

• Input couplings — Contains the necessary information coming from other models, on the native 
model grid, to update the model stale. 

• Model state — Contains the state which is updated by the model. 

• Output couplings — Contains the necessary information to force other models, which arc not 
part of the state, on the native model grid. 

2. Hermes Class — This class contains the coupler utilities to transform from one model representation 
(grid and domain decomposition) into another. A Hermes class also has 3 main data structures: 

• Input couplings — Contains information on a grid of one type and decomposition. 

• Output couplings — Representation ol input intormation on a grid of a second type and 
decomposition. 

• Transform — Operators to transform from one grid and domain decomposition to another.” [ 1 ) 

2.5.4 Evaluation 
Strengths 

GEMS has successfully been applied in climate modeling at NSIPP. Its utilities written in Fortran 90 can be 
used lor developing application codes in Fortran 90. Modular design has been widely used, which reduces 
the complexity of codes. 

Weaknesses 

Since Fortran 90 does not have inheritance and polymorphism, the extensibility and flexibility of the GEMS 
framework is limited. Dynamically allocated storage has been heavily used in GEMS. That is certainly good 
for efficient use of memory during computing. However, it would be formidable for a code written in C/C++ 
to communicate with GEMS since the compiler treatment of dynamically allocated storage is different 
between C/C++ and Fortran 90. 

Summary 


The GEMS framework, which is written in Fortran 90, satisfies the basic functionality requirements for Earth 
System Modeling Framework (ESMF) and has been used in NSIPP’s production code. However, whether 
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GEMS can serve as a community framework is worth exploring since a community Iramework should have 
good flexibility and extensibility in order to satisfy the needs of various organizations. Two object-oriented 
concepts, inheritance and polymorphism, are very important in enabling good flexibility and extensibility. 
Fortran 90 does not have these important features. 

2.5.5 References 

'URL: http://dao.gsfc.nasa.gov/subpages/ofncc-notes.htm] 
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2.6 Flux Coupler 


2.6.1 Introduction 

The Flux Coupler is developed by NCAR ( 1 1. It computes interfacia] (luxes between the various component 
models in the NCAR climate system model (CSM) [2] and distributes these (luxes to all component models 
while insuring the conservation of fluxed quantities. Currently, the Flux Coupler can support four ocean 
models (NCOM at NCAR, POP at LANL, regional pacific, and a dala-only dummy model), three 
Atmosphere models (CCM, dummy model 1, and dummy model II), and land and ice models. 

The Flux Coupler is evolving into a new version (CP15). The new Flux Coupler will implement the following 
new features; 

• Fewer constraints on surface model domains (e.g., allowing shifted pole grids). 

• A more general and flexible method for mapping data between the various grids. 

• A more accurate way of taking one atmosphere’s net solar flux calculation and applying that to ice, 
land, and ocean components with widely varying surface albedos. 

• Miscellaneous efficiency improvements (e.g., with respect to multitasking). 

More detailed information can be found at the ESM Web site |3|. 


The NCAR CSM is not a particular climate model, but a framework tor building and testing various climate 
models lor various applications. In this sense, more than any particular component model, the Flux Coupler 
defines the high-level design of the CSM software. This report is based on review qf documents f ound at the 
CSM Web site ( 1 ], review of Version CPU software, and several teleconferences with our POC, Brian 
Kauffman, and Tom Bettge at NCAR. 

2.6.2 Synopsis 

Prominent information about the Flux Coupler includes the following: 

1 . Community of origin— Flux Coupler is developed by NCAR for the NCAR CSM. 

2. Description It is a separate executable for coupling different climate modeling components such 
as ocean and atmosphere components. 

3. Team — Chief developers are Brian Kauffman for Flux Coupler Version CPI4 and Tom Bettge for 
Flux Coupler Version CPI5. 

4. Maturity A few years; CP14 is the current version and CP15 is under development. 

5. Users — There are a few hundred users in the climate modeling community ranging from national 
laboratories to universities. 

6. Language — The Flux Coupler source code is written almost entirely in standard Fortran 77. Perhaps 
the most notable exception is the use of the library calls “msread” and “mswrite” which rely on 
NCAR’s site-specific Mass Storage System (MSS) for storing large output files. Application model 
components can be written in Fortran 77 and Fortran 90. 

7. Tools/utilities included — The Flux Coupler has Fortran subroutines to map flux fields between 
various model grids, combine like fields from several grids onto one grid, and perform summation 
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and timc-avcragc of quantities. However, those subroutines arc not sufficient] y encapsulated to be 
used by other components such as the ocean components. 

8. Application interface — To use the Flux Coupler, the application model components must follow 
rules set up by the Flux Coupler developers for the data types used in exchanging data between the 
Flux Coupler and the application component. A user cannot modify the Flux Coupler easily to 
satisfy his/her needs. 

9. Documentation — The user's guide for the Flux Coupler is located at [4]. 

10. Associated software — MPI and multitask shared-memory utilities have been used to support 
parallel communication. 

1 1 . Special features — In addition to computing and distributing fluxes, the Flux Coupler also controls 
the execution and time evolution of the complete CSM by controlling the exchange of information 
between the various components. 

2,6.3 Description 

The Flux Coupler performs flux computation and exchange for interacting component models within the 
CSM. This allows the CSM to be broken down into separate components, atmosphere, sea-ice, land, and 
ocean, that are “plugged into” the F,ux Coupler (a.k.a. “driver”). A primary requirement of the Flux Coupler 
is to ensure that conservative properties — such as momentum, heal, and fresh water — arc neither created nor 
destroyed as they arc exchanged between CSM model components. The following characteristics of the Flux 
Coupler were derived by examining the Flux Coupler source code and documentation: 

• The Flux Coupler is a common application for coupling component models in the NCAR CSM. 

There are four components to a complete system: atmosphere, land, ocean, and sea-ice. 

• Each component is required to be connected to the coupler and exchange data through the coupler 
only. 

• Each component model is a separate code with its own computational requirements, such as spatial 
resolution and time step. 

• Individual components can be created, modified, or replaced without necessitating code changes in 
other components. 

• CSM components run as separate executables, communicate via message passing (MPI in particular), 
and processing can be distributed among several computers. 

• The Flux Coupler computes fluxes from component model variables and passes the required fluxes to 
the components while ensuring the conservation of fluxed quantities. 

• The Flux Coupler synchronizes the execution of component models. 

• The Flux Coupler performs grid interpolations of data from component models using different grids. 

• The Flux Coupler uses parallel communication to exchange data with components. 

The high-level design of the Flux Coupler is illustrated in figure 6. The Flux Coupler is written in Fortran 77 
and consists of a main program and subroutines. As shown in figure 6, the Flux Coupler provides data 
exchange services for application models such as the Ocean and Atmosphere components. However, utilities 
in the Flux Coupler cannot be easily used for developing components because they are not sufficiently 
encapsulated. The ellipses in figure 6 represent a group of Fortran 77 subroutines. The purpose of those 
subroutines is to gather, merge, sum, and/or time-average the various component flux fields from various 
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sources and form a sol of complete input fluxes for each component model. A nested loop structure is used 
lor controlling the running sequence ol components. The ocean model communicates with the coupler once 
per outer loop iteration, while the atmosphere, ice, and land models communicate once per inner loop 
itciation. Modular design has been used to lacilitatc implementation ol alternate configurations. A variety of 
time coordination schemes can be, and have been, implemented by rearranging subroutine calls at the highest 
level (within the main program file), requiring a minimal amount of code modification or new code. 



Provide 



Figure 6. High-Level Design at the Flux Coupler 


The Flux Coupler has incorporated a number ol subroutines in the areas of mapping and control. The 
mapping utilities include subroutines tor initialization, bilinear interpolation, merging, verifying acceptable 
component, and masking routines which are listed in table 2. Those subroutines are used in the Flux Coupler; 
they are not available to the application components. 

In the area ol control, there are subroutines to set control flags for stopping the calculations, creating restart 
and history data, and outputting standard diagnostics. These flags are used by the coupler, but are also sent to 
component models to facilitate the coordination ol data sets. The corresponding subroutines are shown in 
table 3. 

Both shared-memory multitasking and message passing (e.g., MPI) have been used in CSM for parallel 
communication. Inside the Flux Coupler, shared-memory multitasking is used. Between the Flux Coupler 
and an application component, MPI is used with a wrapper (msg_recv_r, msg_necv_I, msg_send_r, msg__send _I). 
The application components can use either MPI or shared-memory multitasking. 


Since application model components and the Flux Coupler can be run as an individual executable, the load 
balance among those executables has to be considered. Currently, the load balancing is done manually by 
trial and error. 


2.6.4 Evaluation 
Strengths 

The Flux Coupler has success! ully served the CSM community by providing a utility to plug component 
models into the CSM. It a user follows the rules lor data exchange set up by the Flux Coupler developers, 
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they can test their application components in the environment of CSM through the Flux Coupler without 
changing the Flux Coupler code: however, their interface is not flexible (sec weaknesses). 

Weaknesses 

The Flux Coupler involves many operations, such as flux calculations and execution control, which makes 
maintenance and upgrade difficult. One solution is to divide the multiple lunctionalitics into several 
subroutines and encapsulate the scope of data and functionality in each subroutine. 

The high-level design of Flux Coupler is procedural not object-oriented, which limits its extensibility and 
reusability. The most recent version of the Flux Coupler (CP15) uses Fortran 90 to modularize the code and 
some of the flux calculations have been removed, in favor of allowing the individual components to compute 
the lluxes. 

The original design of the Flux Coupler was intended to provide an independent coupling tool allow ing users 
to test their application component in the CSM environment without modifying the Flux Coupler. In reality, 
users still tend to modify the Flux Coupler code to interface with their application models, rather than simply 
following the rules set up by the Flux Coupler developers. In general, this procedure can result in significant 
configuration management issues with the Flux Coupler code. At the current time, most users do not insert 
new components into the CSM through the Flux Coupler; rather they run simulations with the existing CSM 
components and modify the parameter sets to tailor the computations to their needs. 

Summary 

The Flux Coupler can be run as an independent executable providing coupling for application components. It 
has many users in the climate modeling community. However, its procedural code structure limits the 
flexibility of its interface for application components. In addition, the Flux Coupler contains several separate 
functions (exchanging state variables, calculating and distributing fluxes, controlling the execution of a 
program), which makes the code difficult to maintain and update. 

2.6.5 Reference 

1 http://www.cgd.ucar.edu/csm/mcxlels/cpl/ 

2 http://www.cgd.ucar.edu/csm 

3 http://www.cgd.ucar.edu/csm/models/cpl-ng/ 

4 http://www.cgd.ucar.edU/csm/models/cpl/cpl4.0 
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Table 2. Mapping Utilities 


Subroutine Name 

Purpose 

Assumptions 

niup_inii(Si) 

Initialize and/or update any 
mapping weights or indexes 
for all mapping options 


map_aavg_x 2 o_i nit(ifrac) 

Setup routines for area-averaging 
onto ocean grid 


map_aavg_x2ajnit{ ii'rac) 

Setup routines for area averaging 
onto atmosphere grid. Compute 
weights for ( 1 ) area averaging 
ice and ocean fields onto the 
atmosphere grid (2) merging ice, 
ocean, and land fields (given all 
fields on atmosphere grid) 

Ocean and ice are on identical grids 
Atmosphere and land are on identical grids 
The grids used in this subroutine are “edge'’ (a.k.a. 
“vertex") grids (i.e., in both the x and y [longitude 
and latitude] directions), these are the n+1 
coordinates of grid cell edges which define the area 
associated with the n grid cells 
Any surface area not covered by ocean or ice is 
covered by land 

Further assumptions as per routine map_aavg_rsum: 
-all grids are in degrees (as per spherical/giobal 
grids) 

-all grids are strictly increasing 
-all y grids are in [-90,90] 

-all x grids are periodic, ie. x( 1 )+360=x(nx+l ) 

-all grids contain x=180 (minimum requirement: 
grids overlap) 

map_aavg_rsum 
(xO, y(), nxO, nyO, 
and 

xl, yl, nxl, 
ny I ,nc_max, 
and 

daO, ncl, i 1. jl, dal) 

For each ()-grid cell (i,j), this 
routine computes ncl, i 1 (n), 
j l(n), and dal(n), for n=l,ncl, 
which can then he used to map 
values (via R1EMAN-UM/ 
integral) from the 1-grid 
( which has ccll-cdgc 
coordinates xl and yl ) onto 
the O-grid (which has cell-edge 
coordinates xO and yO) 

Both grids are in degrees (as per spherical/global 
grids) 

All grids are strictly increasing 
Both y grids are in [-90,90] 

Both x grids are periodic, i.e. x( 1 )+360=x(nx) 
The x grids overlap at some point 
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Table 2. Mapping Utilities (continued) 


Subroutine Name 

Purpose 

Assumptions 

map_hilin_init(Xin 
,Yin ,mx,my, 
and 

Xout,Youl,nx,ny, 

and 

w,i 1 ,j 1 ) 

Bilinear interpolation routines. 
For every pair of output grid 
indices (i,j) compute input grid 
indices il(i),jl(j), and four 
weights 

w(ij,l),w(i j,2),w(ij,3),w(i ,j,4), 
such that, FouK i j ) = 
F(Xoul(i),Yout(j)) = 
w(i,j,l)*Fin(il ,jl ) + 

w(i j,2)*Fin(il+l ,jl ) + 

w(i,j,3)*Fin(i l + l,jl + l ) + 
w(i,j,4)*Fin{i 1 ,j1 + l)isthe 
bilinear interpolationof field Fin, 
on grid Xin. Yin, onto field Foul, 
on grid Xout,Yout 

All coordinates (x,y) lie in |(),36()]x| -90,90] 

(i.c., lie on the globe, with units of degrees) 

X and Y grids are strictly increasing 
X(l:mx) is periodic, so that Xin(mx+1) is 
understood to be Xin( 1 ,j)+360.0, likewise, 
Fin(mx+l,j) is understood to be Fin( l,j)if Y( 1 ) > -90 
and/or Y(ny) < +90, 

Yin( (),j) is understood to be -90.0 
Yin(my+l,j) is understood to be +90.0 and 
corresponding north- and south-pole values may 
need to be fabricated by the routine that uses these 
weights and indices, is such cases, 

Fin(i,0) is understood to be a fabricated south-pole 
value, and 

Fin(i,my+1) is understood to be a fabricated north- 
pole value 

merge3(nx,ny, 
wl ,w2,w3, flJ2,f3, 
fsum) 

Merges three quantities on 
the same grid, using 
associated weights to obtain 
a net quantity on the grid. 
Presumably the weights add 
up to 1 .0 


mcrgc2(nx,ny, 
wl,w2, fl,f2, fsum) 

Merges two quantities on the 
same grid, using associated 
weights to obtain a net 
quantity on the grid. 
Presumably the weights add 
up to l .0 


map_check() 

Verify acceptable component 
model domains 

atmosphere and land are on identical grids 
ice and ocean are on identical grids 
ice and ocean have identical domains 

map_maskl() 

set (or reset) mask_l such 
that mask_l(i,j) is non-zero 
iff the coupler will use land 
data at location (ij) Note: 
wm_12a(ij)>0 => coupler 
will use land data at (i,j) 

atmosphere and land are on identical grids 
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Table 3. Control Utilities 


Subroutine Name 

Purpose 

Assumptions 

eontrol_diag( ) 

Set control Hags for the creation of 
runtime diagnostics 

This routine is called once 
at every time step 

eontrol_hist( ) 

Set control (lags for the creation 
and archiving of history Hies 

This routine is called once at 
every time step 

control_rest() 

Set control Hags for restart file 
creation 


control_stop{) 

Set control flags for when the 
integration stops 
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2.7 Distributed Data Broker 


2.7.1 Introduction 

The University of California, Los Angeles (UCLA) Distributed Data Broker (DDB) 1 1 1 results from a 
collaboration between the Department ol Atmospheric Sciences [2 1 at UCLA and the Computer Science 
Department [3] at the University of California. Berkeley (UCB). The DDB, as described on the DDB 
homepage, is a 

"software tool to handle distributed data exchanges between ESM (Earth Science Model) 
components. The DDB differs from conventional coupler tools in that it handles 
communication between two independent ESM components without the need for a central 
communication agent. The DDB has three major components: the Model Communication 
Library (MCL), the Communication Library (CL) and the Data Translation Library (DTL). 

The MCL contains a set of callable routines that are used by the different ESM components 
to register during an ESM run, and perform the exchanges of data. The CL is a set ol routines 
used by the DDB to manage the data exchanges based on the communication libraries 
supported by the available computer platforms. Lastly, the DTL transforms data in a given 
grid domain to the domain of the requesting model. This library will include a number of 
utilities from simple linear interpolation routines to high-order data translation functions. 

The Distributed Data Broker works in a producer-consumer paradigm in which the data 
producers send the data directly to the consumers at given time intervals. The data consumers 
will later receive the data at a rate dictated by its internal computations. [ 1 1 

2.7.2 Synopsis 

Prominent information about DDB includes the following: 

1 . Community of origin — DDB originates in the Atmospheric Science Department at UCLA and the 
Computer Science Department at UCB. 

2. Description — DDB provides a communication mechanism between climate models, enabling them 
to share gridded information without an intermediate agent process. Instead it provides for direct 
communication between components without a separate process. 

3. Team — Past and present members of the DDB team include Tony Drummond |4], C. Roberto 
Mechoso [5|, J.A. Spahr, James Demmel |6|, Howard Robinson [7], and Keith Sklower [8]. Tony 
Drummond, the POC, has recently left UCLA [2] for a new position at NERSC |9|. The DDB team 
is in the process of deciding how the data broker project will be supported. Tony is also a member of 
the Common Component Architecture Forum [10| and the Common Modeling Infrastructure 
Working Group (CMIWG) [ 1 1 1. 

4. Maturity — This project began 1994. 

5. Users — DDB is primarily used within UCLA for coupling climate models in the context of the 
Earth System Model (ESM) |12|. It has been used by the National Partnership for Advanced 
Computational Infrastructure (NPACI) [131 to perform multiscale multiresolution modeling [ 14] 
using Legion [ 15]. There is only one user outside the UCLA group. 

6. Language — DDB is written in C++ and C. The library provides interfaces in both Fortran and C, 
thus allowing it to work with models written in multiple languages. 
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7. Tools/utilities included — DDB comes packaged as three libraries: 

• the Model Communication Library (MCL); 

• the Communication Library (CL); 

• the Data Translation Library (DTL). 

8. Application interface — DDB employs a simple interface scheme in which models register at the 
beginning of the task using a registration broker and they communicate during the task using send or 
receive subroutine calls. The architecture lor this system also seems to be influenced by Common 
Object Request Broker Architecture (CORBA) [ 16] 1 1 7 1. 

9. Documentation — There are several documents specifically about DDB including the home pace 

[ 1 a recent presentation [18] by Mcchoso and others [19J. Much of the information about DDB is 
derived from information about applications which use it, the primary one being the ESM [ 12] at 
UCLA, for which the DDB plays a prominent role [20]. Other publications include articles on 
parallelization and performance of atmospheric chemical tracer models by Demmel and Smith [21 ] 
I22|, the UCLA Atmospheric General Circulation Model (AGCM) [23] by Mechoso et al. (24] [25], 
coupling to an ocean model by Farrara et al. [26], and others [27] [28]. Some fragmentary 
documentation is also available over the Web for project status and results [11]. There is no user/ 
reference manual. Mechoso notes ([18], p. 19) that DDB will soon be available upon request from 
UCLA or at the U.S. National HPCC Software Exchange (NHSE) [29). 

10. Associated software— The DDB can use either PVM [30] or MPI [31 ] libraries. 

1 1 . Performance — I he DDB is designed to replace an equivalent service which runs as a separate 
centralized computational process on an independent processor. In actual tests, as show r n in figure 
10, DDB does indeed perform faster than such a system. 

12. Special features — DDB is targeted towards highly distributed processing. It can work in 
conjunction with Legion [15], a distributed virtual computing system developed at the University of 
Virginia. 

2.7.3 Description 

DDB is designed to operate in a multimodel system where each model distributes its processing across 
multiple processors according to a geographic grid. DDB provides the functionality of an intermediate 
communication agent without the implementation of an intermediate communication agent. 


High-Level Design 

DDB consists ol three libraries — an MCL, a CL, and a DTL. It is designed to serve as a communications 
interface between multiple components in a simulation. 

The Problem DDB Is Designed to Avoid: Centralized Coupling 

DDB is designed to solve problems associated with centralized coupling approaches where, on an N 
processor system with M models, N-l processors are dedicated to model processing and one is designated as 
a communication agent. On such a system, as M (and N) increases, there is an increasing processing burden 
which is cast upon the single communication process. Thus the potential exists to have N- 1 processors 
waiting while an overloaded single processor handles all of the communication between them. This situation 
can be aggravated in cases where different models have different grid sizes where a geographic region on one 
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model (corresponding to a single processor) would be, tor a different model with a diticrent grid, associated 
with multiple processors that all own corresponding parts of the subgrid. In such a situation, coupling data 
from a region on one model to another model might involve numerous processors, all of which can be 
affected by a communication bottleneck. Thus, this method of dividing the processors into model and 
communication processor tasks has performance risks. 


The DDB Solution: Distributed Coupling 

The DDB approach provides the appearance of a central coupler with the implementation of a distributed 
coupler. In this case on an N processor system all N processors can be dedicated to model processing and 
when communication is required, it is handled on a point-to-point basis with both processors sharing the 
communication processing burden. Thus this is a solution which is more appropriate tor distributed 
processing and can scale more effectively. DDB can be shown diagrammatically as an intermediate agent 
between all models but functionally it is implemented as a function call which operates in each processor’s 
process space. 

DDB Implementation 

The DDB provides the functionality of a communication agent between model processes without the agent 
being implemented as a separate process. Figure 7 illustrates how DDB is considered in the context ol the 
ESM [ 12], incorporating the following models: 

• the UCLA AGCM [23]; 

• the Oceanic General Circulation Model (OGCM)[32]; 

• the UCLA Atmospheric Chemical Model (ACM) [33]. 

The models communicate using DDB as an intermediate agent but DDB docs not run as a separate task. The 
diagram shows all models using function call API, implemented as part ol the DDB library. 


Schematic Representation of the UCLA Earth System Model 



Figure 7. DDB as Part of the ESM 
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As described on the DDB home page, the DDB operates in two phases. In the first phase, it employs a 
registration broker to collect model information. In the second phase, it functions as a communication 
intermediary between the other models. Each of these phases are explained below. 

Model Registration 

To use the DDB each process must register with it at the beginning of each run. A Registration Broker (RB), 
implemented as subroutine calls, is used to collect information on each model which includes both the 
information it can supply and the information it requires from other models. This information specifically 
incorporates information about grids. The RB returns to the model a set of information which can be used in 
connection with the later communication calls. After registration, each process has enough information to 
communicate with the other models. This procedure is illustrated in figure 8. 

MCL Registration 


Model A 


MCLMetaRegister 


Model Ctrl process 
registers subdomain 
information 
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data production and 
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between processes 
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from subdomains 
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MCLRegisterConsume 


Ctrl process registers 
data production and 
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MCLRegisterProduce 
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Model Ctrl process 
registers subdomain 
Information 


MCLMetaRegister 


Figure 8. DDB Registration [1] 


Inter-Model Communication 

During the rest of the run, models communicate using MCLGctData and MCLSendData function calls as 
illustrated in figure 9. As described by Drummond: 


“In figure 9, PaO requests data from Model B. The region in the Model B’s domain that 
corresponds to the subdomain being worked by PaO is highlighted by the orange rectangle. 
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This rectangle covers the subdomains being worked by Pb(), Pbl, Pb2, Pb5, Pb6, Pb7. 
Therefore, a simple call to MCLGctData is translated in to 6 reccivcs-opcralions from each 
of the Model B processes. Likewise, a single MCLScndDala from each of the Model B 
processes will get translated into one or more send-operations to Model A processes. Alter 
all the data is received by PaO, it pastes the different information received and translates the 
data to its own grid units." [ 1 ] 

The communication takes place above the MPI |3 1 1 level. Each send may be translated into multiple 
MPI calls. 

MCL Send and Receive Data 


MCLGetData Model A 



MCLSendData 

Figure 9. Inter-Model Communication [1] 
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Figure 10* Comparison of Centralized Coupling Versus DDB Coupling as a 
Function of Number of T3E Nodes f 1] 


Performance 


Performance results for DDB versus a centralized coupling implementation are shown in figure 10. The 
figure shows a decrease in time (seconds per simulation day) ranging from 7.5 percent for 250 nodes to 12 
percent for 500 nodes and 7.5 percent for 800 nodes. 

2.7.4 Evaluation 

According to Tony Drummond, who has investigated other coupling software similar to DDB and presented 
concise comparison results [341 to CMIWG [11], the DDB has characteristics in common with the NCAR 
Flux Coupler [35], CSU Flux Coupler [36], CERFACS Coupler (OASIS) [37] [38], and the Max Suarez 
Coupler [39] in that it provides a modular interface between different numerical models. DDB is unique in its 
registration and distributed coupling approach. 


According to the definition, DDB is not a framework as “a framework is a reusable, semi-complete 
application that can be specialized to produce custom applications,” ([40], p. 4), but it has the potential to 
play an important role in any framework in that it efficiently “describes the interface of each object and the 
How of control between them,” ([40J, p. 4). 

Strengths 

The strengths of the DDB are the following: 

I . Library — It is packaged as a library, which means it can be easily incorporated into other 

framework solutions. 
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2. Language— It has dual Fortran and C interlaces which cases integration into larger frameworks. 

3. Simple API — The API for registering models and communicating between them is simple, a sign of 
good design. 

4. Distributed — DDB can work in distributed environments without tying up another processor. 

5. Registration— The DDB uses a registration process for the models, a procedure which should be 
incorporated into any framework. 

6. Proof of concept — DDB shows that this concept works. 

Weaknesses 

The weaknesses of DDB are as follows: 

1 . Few users — It has a small user community and therefore may not be as robust as other couplers 
with more users. OASIS [38], on the other hand, has at least 15 user groups as o! 1997. 

2. Lack of documentation — The lack ot a user manual, reterencc manual, or other docu mentation is a 
serious deficiency, particularly compared to other couplers which provide User’s Guides, such as 
OASIS [41 1 and NCAR FC [42]. 

3. Uncertain support— The DDB team does not seem to offer the level of support of Cactus [43 1 or 
ROOT [43]. DDB seems mainly to be an internal project. The issue of support becomes more 
uncertain with the loss of Tony Drummond [4] to NERSC [9). 

Summary 

The UCLA DDB is not, by itself, a complete framework. Nevertheless, the task it seeks to accomplish, 
exchange of gridded data, represents a crucial part of any framework solution. DDB seeks to fill the role of 
coupler using a distributed technique. This technique works and provides some performance gains in 
comparison to centralized coupling, which makes it worthy ot close examination tor the underlying 
technology because the general design concept is excellent. At the same time the number ol users is small, 
there is little specific documentation and the level of future support from the team appears uncertain. 
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2.8 Other Frameworks 


This section examines three other Iramcworks — ROOT, PAWS, and ALICE — in somewhat less detail than 

the prior six. 

2.8.1 ROOT 

2.8.1. 1 Introduction 

ROOT arises from the European Organization for Nuclear Research (CERN) f 1]. ROOT is described in the 

Executive Summary ot the mission statement on the ROOT home page as follows: 

The ROOT system provides a set of OO frameworks with all the functionality needed to 
handle and analyze large amounts of data in a very efficient way. Having the data defined as 
a set of objects, specialized storage methods are used to get direct access to the separate 
attributes of the selected objects, without having to touch the bulk of the data. Included are 
histogramming methods in 1, 2 and 3 dimensions, curve fitting, function evaluation, 
minimization, graphics and visualization classes to allow the easy setup of an analysis 
system that can query and process the data interactively or in batch mode. 

Thanks to the built-in CIN F C++ interpreter the command language, the scripting, or 
macro, language and the programming language are all C++. The interpreter allows for fast 
prototyping of the macros since it removes the time-consuming compile/link cycle. It also 
provides a good environment to learn C++. If more performance is needed the interactively 
developed macros can be compiled using a C++ compiler. 

The system has been designed in such a way that it can query its databases in parallel on 
MPP machines or on clusters of workstations or high-end PC’s. ROOT is an open system that 
can be dynamically extended by linking external libraries. This makes ROOT a premier 
platform on which to build data acquisition, simulation and data analysis systems ” [2] 

2.8. 1.2 Synopsis 

Prominent information about ROOT includes the following: 

1 . Community of origin — ROOT arises from CERN [ 1 ] in the particle physics community. It was 
designed to support data analysis for the Large Hadron Collider where the expected amount of data 
produced exceeded 1 million GB (1 PB) per year. 

2. Description — “ROOT is a system for large-scale data analysis and data mining. It is being 
developed for the analysis of particle physics data, but it can be equally well used in other fields 
where large amounts ol data need to be processed.” [3] 

3. Team Team members include Rene Brun [4] and Fons Rademakers [51. 

4. Maturity — this project has been functioning since 1994 and is still active. Version 2.25/02 has 
recently been released. 

5. Users— According to a paper by Rademakers and Brun in 1998, “More than 16,000 copies of the 
ROOT binaries have been downloaded from [ the J Web site, about 700 people have registered as 
ROOT users, and the Web site gets more than 150,000 hits per month. ROOT is currently being used 
in many dilferent fields: physics, astronomy, biology, genetics, finance, insurance, pharmaceutics. 
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etc." ( 3 ] Since that time the number of distributed binaries has increased to more than 75,000. The 
ROOT home page also lists at least 37 applications using ROOT [6|. Thus, ROOT is a broadly 
disseminated development framework. 

6. Language — C++ 

7. Tools/utilities included — According to Brun [7|, class categories include basic ROOT classes, 
container classes, histogram and minimization classes, tree and n-tuple classes, two-dimensional 
graphics classes, three-dimensional and detector geometry classes. Graphical User Interlace (GUI) 
classes, interactive interlace classes, the C++ interpreter |8|, the operating system interlace, 
networking classes, and documentation classes. 

8. Application interface — The application interlace consists oi the object-oriented class structures. 

The interfaces arc well thought-out and extremely well documented. 

9. Documentation — The ROOT Web site has an enormous amount of documentation including 5 1 
tutorials |9], 37 applications examples |6], a reference guide ( 101, and numerous other documents. 

An interesting presentation by Federico Carminati describes the migration by CERN to C++/OO 
away from Fortran [11]. On page 6, he discusses how this choice was made: 

• Migrate immediately to C++. 

• Adopt the ROOT framework. 

• Allow use of Fortran and C++. 

• Impose a single framework. 

He also discusses specific policies adopted regarding programming styles in C++ (p. 15), support for 
a Bazaar [ I 21 model of development (p. 16), and lessons learned in migrating scientists familiar w'ith 
Fortran to a new environment (p. 25). 

10. Associated software — The C++ interpreter [8] is an associated software package that has been 
tightly integrated into the ROOT system. 

11. Performance — Unknown. 

12. Special features — According to the developers the main tcalurcs ol ROOT include the runtime type 
information system 113], the object I/O system 1 14], and automatic documentation generation 115]: 

• The runtime type information system 1 1 3 1 is a memory resident dictionary that, at runtime, 
maintains information on all objects including lists ot all global functions, all global variables, 
all classes, data member descriptions of the classes, and class member Junctions. This 
centralized information source is a powerful capability not implemented in any other framework 
surveyed. 

• The object I/O system 1 14] includes a set of classes to support I/O to/from machine independent 
files. It is “designed to be particularly efficient for objects manipulated by physicists: 
histograms, n-tuplcs, trees, and events.” 

• ROOT provides for automatic documentation generation [15] using a set of documentation 
classes which “allows the creation of hyperized (in HTML formal) C++ header and source files, 
inheritance trees, class indices, macros, and session transcripts. Thanks to this facility almost 
everything in the ROOT system can be automatically documented and cross-referenced.” This is 
a tremendously useful capability not duplicated in any other framework. 
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• ROOT provides an excellent example of an object hierarchy. It has 310 classes grouped in about 
24 frameworks divided into 14 categories |7). 

• ROOT uses a C interpreter (CINT) [8|, for the scripting language. This is a great example of 
how a scripting language should be associated w ith a framework, though it is not necessarily 
obvious that interpreted C or C++ represents the best choice. 


2.8. 1.3 Description 

ROOT is primarily oriented towards data analysis instead of simulation. The distribution in focus between 
event generation, detector simulation, event reconstruction, data acquisition, and data analysis is shown in 
figure 1 1 . 

Unlike many frameworks, ROOT has a true class inheritance hierarchy. The ROOT schema organization is 
loosely illustrated in figure 12. Basic classes in an inheritance hierarchy arc shown in figure 13. 

Figure 14 illustrates the ROOT environment and tools. 



Figure 11. The Primary Speciality of the ROOT Framework is 
Fvent Reconstruction, Data Acquisition, and Analysis |2| 
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Figure 12. ROOT Schema |2] 
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Figure 14. ROOT Environment and Tools [2| 
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2.8. 1.4 Evaluation 


ROOT is an excellent example of a scientific object-oriented framework. At the current time it is not 
specifically suited to handling models for the climate community. 

Summary 

ROOT exceeds all other surveyed frameworks in terms of number of users, maturity, extent of 
documentation, and excellence in object-oriented design. Is worthy of close inspection and emulation by the 
climate community. The I act that a scientific organization such as CERN has migrated from Fortran to C++ 
using ROOT provides lessons regarding options available to the climate community. 
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2.8.2 PAWS 


2.8.2. 1 Introduction 

The PAWS originates in the Advanced Computing Laboratory (ACL) 1 1 1 at LANL |2| with some 
collaboration from the NEXUS project |3| at ANL |4| and the POOMA project [5| at LANL |2|. As 
described on the home page: 

“PAWS (Parallel Application Workspace) is a software infrastructure for use in connecting 
separate parallel applications w ithin a component-like model. A central PAWS Controller 
coordinates the linking of serial or parallel applications across a network to allow them to 
share parallel data structures such as multidimensional arrays. Applications use the PAWS 
API to indicate which data structures arc to be shared and at what points the data arc ready to 
be sent or received. PAWS implements a general parallel data descriptor, and automatically 
carries out parallel layout remapping when necessary. Connections can be dynamically 
established and dropped, and can use multiple data transfer pathways between applications. 

PAWS uses the NEXUS [3| communication library and is independent of the application's 
parallel communication mechanism.” [6] 

2.5.2.2 Synopsis 

Prominent information about PAWS includes the following: 

1 . Community of origin — PAWS originates at ACL ( 1 1. 

2. Description — Provides a mechanism to connect separate parallel applications. 

3. Team— Team members include Peter Beckman [7], Patricia Fascl [ 8 ], Bill Humphery, Sue 
Mniszewski, and Teresa Roberts. 

4. Maturity — This package does not seem to be very mature. There arc few papers and lew users. 

5. Users — There arc no know n users outside of the PAWS team. 

6. Language — C++ 

7. Tools/utilities included — PAWS is considered an ACTS [9| toolkit component. 

8. Application interface — PAWS has a Fortran and C interlace. 

9. Documentation — There are several documents specifically about PAWS including a users guide 

[ 10), a programmers manual [ 1 1 1, a programmers reference [ 12|, a conference publication [ 13|, and 
an overview presentation 1 14|. The home page also provides a project overview and technical 
summary. 

10. Associated software — PAWS employs the NEXUS |3) communication mechanism for its message 
passing substrate, and is currently in the process of adding the ability for PAWS applications to work 
within the GLOBUS 1 15 1 metacompuling environment. Initial test applications use the POOMA |5| 
framework. The POOMA framework includes an interface that uses PAWS to alknv all POOMA 
multidimensional array objects to be shared with other programs. 
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I 1. Performance — Performance has not been evaluated. 


12. Special Features — TBD 
2.S.2.3 Description 

The PAWS framework concept has something in common with the role played by the UCLA DDB 1 16| and 
the NCAR Flux Coupler [17): the sharing of information between running modules. PAWS is primarily 
concerned with the sharing ol scalars, multidimensional lixcd-si/e arrays, and multidimensional varying- 
si/ed arrays. Like the (lux coupler, PAWS has a separate controller process. Unlike the flux coupler, it has a 
scripting language based on Tel [ 1K|. 

Figure 15 shows functional segments and interfaces of the PAWS framework. Figure 16 show's the 
relationship between the controller and other applications. 



Figure 15. Breakdown of PAWS Programming Interface and Controller 1 10 1 



Figure 16. The PAWS Controller Interacting with Several Applications 1 10 1 
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2.S.2.4 Evaluation 


PAWS docs not perform interpolation functions or mapping to coordinate systems like the Flux Coupler or 
data broker. For this reason it is probably not suited for specific tasks needed by the climate community. 


Summary 

PAWS does not seem mature enough at this time to merit a great deal ot consideration by the climate 
community. Nevertheless, the concepts of hav ing a controller associated with a scripting language and that ol 
registration of the applications with the controller, are usetul examples ol capabilities that a tiamcwoik 
should have. 
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2.8.3 ALICE 


2.8.3. 1 Introduction 

Advanced Large-Scale integrated Computational Environment (ALICE) originates in the Mathematics and 
Computer Science Division of ANL [ 1 ]. As described on the homepage: 


I he goal ol the ALICE (Advanced Large-Scale Integrated Computational Environment) 

Piojcel is to eliminate barriers in using independently developed software components in the 
construction of high-performance numerical applications. We believe this will lay the 
groundwork lor widespread exploitation of lerallop-scale computational resources and for 
new scientific insights. 

The ALICE project, a collaborative cl fort among researchers in the Mathematics and 
Computer Science Division ol Argonnc National Laboratory, has grown out of our long 
tradition ot expertise in high-perl ormance sottwarc. This experience has demonstrated the 
benelits ol encapsulating numerical and parallel computing expertise in user-ready tools. 

However, the complexity ol today’s large-scale scientific simulations often necessitates the 
combined use ol multiple software packages to address areas such as mesh manipulations, 
numerical solution of partial differential equations, optimization, sensitivity analysis, and 
visualization. While efficient and robust tools exist, combining them remains difficult 
because of data management and interoperability problems. ALICE research focuses on 

1 . developing low-overhead mechanisms for integrating extensible software for scientific problem 
solving: and, 

2. building component-based toolkits that encapsulate expert knowledge in numerical algorithms 
and parallel computing. 

ALICE development is motivated by a range ol large-scale scientific applications that 
ensure the relevance and practicality ol our design. Our approach supports both new and 
legacy applications, thereby enabling scientists to reuse legacy kernels and to program in the 
style ol most comlorl to them, lor example, traditional Fortran development or more object- 
oriented paradigms. In addition, we are actively engaged in dialogues within the DOE 
Common Component Architecture Forum [2] concerning component-based software 
interoperability throughout the DOE high-performance computing community.” [3] 

2.8.3.2 Synopsis 

Prominent information about ALICE includes the following: 

1 Community of origin— ALICE originates at ANL [ 1 ]. 

2. Description — A set ot tools for building scientific applications. 

3. Team — Primary investigators include Ibrahima Ba, Satish Balay [4], Steve Benson, Anthony Chan, 
Paul Fischer, Lori Frcitag, Bill Gropp |5|, Paul Hovland, Jeff Linderoth, Rusty Lusk, Lois Curfman 
Melnnes, Jorge Mor, Lucas Roh, Barry Smith, Deb Swider, Rajecv Thakur, Henry Tufo, Steve 
Wright, and Golbon Zakcri. 

4. Maturity As a whole ALICE does not seem mature because of sparse documentation. 
Individually, the tools may be just fine. 
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5. Users - projects using ALICE include 

• The Center on Astrophysical Thermonuclear Flashes (ASCI ASAP Center) 

• Multi-Model Multi-Domain Computational Methods in Aerodynamics and Acoustics (NSF 
Multidisciplinary Challenge Project) 

• Massive Crystallographic and Microtomographic Structural Problems (DOE Grand Challenge 
Project) 

6. Language — ALICE has a multilanguage architecture of object-oriented libraries that arc usable by 
Fortran, C, C++, and Java. 

7. Tools/utilities included — The core of the ALICE infrastructure is formed by the following tools: 

• Automatic Differentiation: AD1C and AD1FOR — ADIFOR and ADIC arc source translators that 
augment Fortran 77 and C programs with derivative computations. 

• High-performance portable I/O: ROMIO — A high performance, potable implementation ol 
MPI-IO. 

• Message-passing tools: MPICH [6]— A specification for the user interface to message passing 
libraries for parallel computers. 

• Optimization: MINPACK and NEOS— Libraries of advanced optimization software. 

• PDE and numerical linear algebra software: PETSC [7| and BlockSolvc95 — Function libraries. 

• Unstructured Mesh Computations: SUMAA3D [8| — A Iramework lor parallel unstructured 
mesh computations. 

• Computational Steering— a new project to develop a computational steering system based on 
hierarchical adaptive analysis, parallel computing, and immersive visualization. 

• Futures Lab: [91— Explores, develops, and prototypes next-generation computing and 
communications infrastructure systems. 

• Distributed Supercomputing Tools: GLOBUS [ 1()|. 

8. Application Interface— With respect to interfacing, the ALICE home page (3| provides the 
following information: 

“Critical is the design of interfaces to handle the interconnections among components. 

Clearly, no single mechanism will solve all problems. ALICE uses multiple layers to provide 
both coarse-grained functionality in connecting applications together and fine-grained 
functionality for high-performance data sharing. Key design features arc: 

• a numerical object interface based on mathematical abstractions (e.g., the interface 
specifies classes of linear and nonlinear solvers as opposed to particular algorithms and 
data structures), and 

• a common interface specification among components that may use various underlying 
implementations to address portability and performance issues." 


9. Documentation— The sole document for ALICE is an overview slide presentation [ 1 1 1. Individual 
documentation exists lor some ol the components and is available trom the ALICE homepage [3]. 

10. Associated software — The whole framework is a collection ot tools. 
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I 1 . Performance — Performance can only he evaluated for indiv idual tools. 

1 2. Special features — This Iramcwork is oriented towards interoperability between a diverse collection 
of tools. 

2.S.3.3 Description 

Most ol the information available on ALICE is conceptual and high level, though some detailed information 
is available for the individual components. Figure 17 shows the ALICE infrastructure concept where ALICE 
components are used by applications and in turn are based on lower-level components. Figure 18 illustrates 
the type of components that can be potentially integrated into the ALICE framework. Figure \9 illustrates the 
relationship between ALICE components and toolkits. Figure 20 illustrates the ALICE vision of how 
• components would work together in advanced applications. 
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Figure 17. ALICE Infrastructure Concept (from [11 1, p. 15) 
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Figure 18. ALICE Concept of Computational Components (from (11], p. 8) 
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2 .8.3.4 Evaluation 


There is not sufficient information on ALICE to make a usclul evaluation. Based on the amount of available 
information, this project does not seem to have matured yet. 


Summary 


AL1C E is use I ul in that it has developed a high-level concept ol how a variety of components could work 
together in a framework. More information is required before this framework could be recommended for the 
climate community. 
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3.0 Discussion and Recommendations 

3.1 Motivation for Earth Science Modeling Framework (ESMF) 

The primary motivation behind the ESMf [ 1 1 is to provide a community software inti asti uctuie, in the spiiit 
of the report 1 1 ) issued by the Common Modeling Infrastructure Working Group (CMIWG) |2| and in the 
spirit of the Cooperative Agreement Notice (CAN) |3|, and provide an infrastructure which 

1 . provides a common software base for the entire community; 

2. specifies systems and methods by which models are developed but not the models themselves; 

3. focuses more upon the interactions and collaborations (|4|, p. 4) between the models and less upon 
the implementation of the models; 

4. provides efficient and flexible means of communication and coupling between models: 

5 . provides the means, including tools and utilities, to develop focused core models [ 1 1 which can be 
used as standard models w ithin the community; 

6. provides sufficient flexibility to allow' the community to develop variations and alternatives of core 
models as needed for research, experimentation, and operations; 

7. is designed so that modeling advances can be primarily determined by community consensus and 
scientific merit instead of softw are favoritism or rigidity or the disposition of the organization 
responsible for supporting the framework; 

8. can support both operational and research modeling efforts; 

9. provides technology-transfer mechanisms to allow research model capabilities to migrate to the 
operational centers; 

|(). caters specifically to a community skilled in the Fortran language, recognizing and supporting 
Fortran as a valued language for representing functional mathematical relationships; 

1 1 . provides a means to not only reuse code that has been developed in the past, but to also reuse code 
that will be developed in the future; 

12. provides a bridge to new' computing technologies, allowing researchers to access new' capabilities 
without requiring major modifications to established and trusted code; 

13. fosters documentation of individual models, communication between researchers, and cooperation 
among organizations. 

The ESMF does all this with lower costs, shorter development times, more simplicity, higher reliability, and 
faster speeds, even as it is used with more complex computing architectures and with more models written by 
more participants from more diverse backgrounds to solve increasingly difficult problems. 

3.2 Current Status of Existing Surveyed Frameworks 

In the previously established context of what an ideal framework should provide, the surveyed lrameworks 
can be grouped into several categories: 
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1 . Couplers lor Earth science application components. 

2. Community level application frameworks. 

3. Object-oriented framcworks/toolkils. 

4. Earth science frameworks. 

3.2.1 Couplers for Earth Science Application Components 
A Coupler Is Not Really a Framework, but Plays a Crucial Role 

Couplers do not have the breadth to be considered as framework by themselves. Nevertheless, by providing 
crucial inlorrnation coupling services between models, couplers arc found near the core of any framework 
design concept because they represent the primary collaboration and communication mechanism between the 
models and their limitations and strengths primarily determine what can and cannot be done within the 
framework. An inappropriately chosen coupling mechanism can force users outside the framework to obtain 
either speed or enhanced communication, thus defeating the framework concept in its entirety, by promoting 
the proliferation of additional interfaces w ithin the community. On the other hand, a strategically chosen 
coupler can relieve researchers ol communication difficulties and allow them to focus on more scientifically 
diverse problems, thus promoting the use ol the framework in the spirit of community collaboration. 

Two Couplers Considered: DDB and Flux Coupler 

Two couplers considered in this survey include the UCLA Distributed Data Broker (DDB) [5] and the 
National Center for Atmospheric Research Flux Coupler [6) as only two of the many [7] couplers that could 
potentially be considered. 

DDB has a Better Design 

Of the two couplers, DDB clearly stands out as being designed upon better principles than the Ffux Coupler. 
DDB is primarily designed as a coupler and only a coupler, whereas the Flux Coupler design incorporates 
coupling, tlux computation and control into a single unit which constitutes a delect with serious Jong term 
architectural consequences for the entire framework. 


Why the Flux Coupler Design Is Flawed 

To understand w hy the FC design is flawed it is important to properly understand the role of a framework, 
such as in the context discussed by Fayad and Johnson (|4|, p. 4). A framework provides at least two major 
services: 

1 . It describes the interface between objects. 

2. It provides a capability known as inversion of control (p. 5 ) which is a supervisory service, at a high 
level, for all of the participant objects in the simulation. 


Both of these capabilities are needed in any major simulation framework. It is understandable, in the absence 
of a lruc simulation framework and because the functions arc somewhat related, that the designers of the Flux 
Couplej would combine them into a single unit. Notwithstanding this choice, the issues of communication 
between participant models and control of the same models are two separate functions and should be 
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properly be designed and implemented as two different capabilities in separate modules: a coupling 
capability and a control capability. 

The combination of these two distinct capabilities in a single unit, which in addition to combining the two 
separate principles also executes as a single and separate computing process, is a choice that not only 
complicates the coupling process but also works against the implementation ol a true high-level simulation 
controller, either by the framework developers or by individual researchers. 

If used in a community framework, the Flux Coupler, as a separate entity combining control and 
communication functions, has the potential to be subject to continued modification and upgrades as 
researchers reejuire either improved control, improved communication, ot additional coupling with new and 
different models. Because of these requirements, the Flux Coupler will have to undergo continual changes as 
researchers from the community work closely with NCAR to get it to incorporate new capabilities to talk to 
new models they w ish to use or old ones they wash to modify. Each change may impact the interlace, which 
in turn may impact every member of the community who has implemented a model using that inleilace. This 
may in turn drive the researchers to find ways of communicating between models without using the Flux 
Coupler, so they arc not subject to a changing interlace lor a crucial service. This will in turn dcleat the entiic 
framework concept. 

These, then, arc the potential long-term consequences resulting Irom the Haw in the Flux Couplci design 
which combines communication with control in a single module. 

DDB Does Not Have the Flux Coupler Flaws 

The DDB, on the other hand, has none of the Flux Coupler Haws. It is designed primarily as a general- 
purpose coupler upon sound principles similar to those used by Common Object Request Broker Architecture 
(CORBA) 1 8 1, an object-oriented communication architecture. Furthermore, it does not run as a separate 
process but executes in the process space of the communicating models, which is an advantage tor distributed 
system because it reduces bottlenecks. The interface is simple: Models register at the beginning and 
communicate with each other directly as needed using straightforward calls. DDB is implemented as three 
libraries and has C and Fortran interlaces, thus it can integrate nicely with almost any tramework 
architecture. 

The Interface Is the Most Crucial Part of a Framework 

For a function as crucial to a framework as communication and coupling between participant objects, nothing 
is more important than a simple, flexible, and stable interlace. It docs not matter, hypothetically speaking, ii 
DDB runs slower, has fewer interpolation options, supports fewer message-passing libraries, docs not support 
as many grid systems, and is scientifically interior in every way to Flux Coupler or any other coupling 
system. 

In such eases, DDB performance could be improved, interpolation options could be added, more message 
passing libraries could be supported, as could additional grid systems, and so on. In each and every ease 
these changes would hardly impact the interface. This, then, is the most important feature because it 
promotes the use of the community framework. 

Disadvantages Associated with DDB 

DDB is developed by the University of California, Los Angeles (UCLA ) group and has only one user outside 
the group. Tony Drummond, the lead developer, has left the group lor a new position at NERSC. The issue ol 
long-term support is currently uncertain. The DDB documentation is sparse. DDB docs not have the bcnelit 
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ol hav ing been used by many users, so il may potentially have bugs. These are all negatives which are 
completely unrelated to the strength ol the DDB design but which need to be overcome. 

Advantages Associated with Flux Coupler 

Flux Coupler is available from NCAR, a larger organization with a Web site containing a large amount of 
high-quality documentation. Flux Coupler is in its fourth release and is probably quite stable and has more 
users than DDB. These are all positives that, unfortunately, cannot compensate for the weakness of the Flux 
Coupler design. 

3.2.2 Community Level Application Frameworks 
Two of the Contenders: Cactus and ROOT 

There are two majoi open source trameworks which quality lully as true general-purpose application 
frameworks: Cactus [9) and ROOT 1 1()|. 

ROOT Is Elegant and Is in C++ 

ROOT is an elegant example ot the kinds of services that can be provided by a true object-oriented 
framework. The elements of the framework arc well designed and, as a result of the common inheritance tree, 
have high consistency that provides numerous capabilities with simple interfaces. One of the strongest points' 
is a line documentation system, a critical component for any scientific community. ROOT has users 
numbering in the thousands. It is written in C++. It has almost no climate-specific features. These would all 
have to be added by the climate community as well as a means to effectively integrate Fortran modules into a 
C++ framework. Whether or not ROOT is actually used in an eventual climate framework, its design should 
be carefully studied so that as many features as possible can be imitated. 

Cactus Is Practical for a Fortran Community 

Cactus is a community level framework but has different strengths than ROOT. Cactus supplies object- 
oriented features like inheritance by simultaneously attaching them on top of both Fortran and C. This is no 
small leaf, considering the differences between the two languages. Cactus accomplishes this task, though 
with an awkwardness which is inherent to this type of approach. 

Cactus provides a high-level development environment with numerous interfaces to Fortran 77, Fortran 90, 

C, and numerous other systems. It can produce executables that can run on laptops or parallel computers. It 
carefully defines the interlace by the control structure (flesh) and modules (thorns) and implements 
communication between modules effectively. It also provides powerful supervisory capabilities, such as 
inspecting a running simulation via a Web browser 1 1 1 1. This is an enticing example of a type of new service 
which can be effectively implemented by powerful frameworks such as ROOT or Cactus without requiring 
changes to legacy code. Cactus has many users, a strong support group, is mature and stable, has a lot of 
documentation, but no climate-specific support features. 

If the Climate Community Is Not Willing to Migrate to C++ then Cactus Is a Better Choice 

In a contest between Cactus and ROOT for choice of framework for the climate community. Cactus would 
clearly be supcrioi because of its explicit support for Fortran modules. Nevertheless, climate models would 
have to be modified at the highest levels to conform to the Cactus interface. Using Cactus requires the full 
commitment of the community as once the code is modified, at a high-level, for the Cactus interface, it 
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cannot he independently compiled with a Fortran compiler as it was be I ore. It is not clcai that the Cactus 
interface would work directly for coupling between models in a manner similar to DDB or Flux Coupler. It is 
likely that some cooperation with the Cactus team would be necessary to incorporate these types ot leaturcs. 
The Cactus team has a good track record of cooperating w ith many institutions. 


3.2.3 Object-Oriented Frameworks/Toolkits 

Two C++ Contenders: Parallel Object-Oriented Methods and Applications (POOMA) and Overture 

POOMA [ 121 and Overture 1 13] are two frameworks which arc written in C++ and are used for Partial 
Differential Equations (PDE) problems. Both frameworks offer a user the opportunity to build an application 
using components from the framework: a collection of C++ classes. Recognizing that migrating to C++ trom 
Fortran is a difficult task, each of them takes different approaches. 


POOMA Uses Expression Templates to Achieve Speed Gains 

POOMA takes the approach of using the standard C++ STL libraries and implementing high-speed classes 
using an expression template technique [14] [15] [16], Regardless ol the speed benefits, the bcnchts are 
achieved by the introduction of a type of complexity that will impede researchers migrating to C++ from 
Fortran. This technique is also subject to larger compile times due to the use of a technique that coerces the 
compiler into performing optimizations lor which it is not ideally designed. 

The POOMA approach was used to support Parallel Application Workspace (PAWS) 1 17], another framework 
for connecting parallel applications that was briefly considered in this survey. The POOMA team has 
virtually disbanded recently and future support is questionable, although it is likely that their sponsor will 
continue to need POOMA capabilities. This fact, in addition to the complexity oflhe approach, suggests that 
this framework should not be considered tor inclusion in a climate community Iramcwork. 


Overture Uses a Preprocessor to Achieve Speed Gains 

Like POOMA, Overture seeks to obtain speed improvements in C++. Overture uses a dittereni approach than 
POOMA. First, it developed a powerful array package, A++/P++ [ 18|. which is available separately and may 
be of interest to the climate community. Second, building on these classes it implements a variety ot 
functions. Third, it uses a souree-to-source translator, SAGE++ [ 19] [201, to improve execution speed instead 
of expression templates. This is a more slraighllorward procedure though it introduces an extia step into 
compilation. Thus, independent of Overture itself, there arc at least two components of Overture, A++/P++ 
and SAGE++, which can be of use to the climate community framework. A disadvantage of Overture is that 
the team has their own research work and cannot as easily support independent requests lor soltware 
changes, as could the POOMA team (before it lost most of its people). 


Overture Approach Is More Practical 

In either case, the strict use of C++ and the current situations w ith both POOMA and Overture make it 
unlikely that the climate community can adopt either framework completely, though parts ol' Overture could 
be used. These issues arc still separate from the Fortran/C++ speed issue in which C++ can still stand 
improvement. The importance of this issue needs to be addressed separately by the climate community given 
that Fortran is faster but C++ is better designed for constructing frameworks, possibly requiring source-code 
translators to merge the two should a joint approach be taken. 
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3.2.4 Earth Science Frameworks 


Two Contenders: Goddard Earth Modeling System (GEMS) and Flexible Modeling System (FMS) 

Two frameworks arising directly in (he Earth science community arc GEMS and FMS 1 2 1 1. (Nolc: GFDL’s 
EMS was not investigated as part of this Task 4 activity, but was examined in the Task 2 survey of current 
Earth system modeling applications, as was GEMS.) These frameworks have neither the elegance of ROOT, 
the lull-service development features of Cactus, nor the object-oriented characteristics of POOMA or 
Overture. What they do have is a direct orientation towards the primary function of the climate community: 
climate modeling. 


Both Are Specifically Geared to Climate Issues 

Both GEMS and FMS are Fortran 90 implementations and both address model coupling issues. FMS has an 
excellent amount ol documentation and GEMS has little. Both arc currently used for modeling; problems. 
Both ol these frameworks have an inherent weakness in that they do not use a true object-oriented language. 

3.3 Summary 


This section contains overall observations coming out of the survey as well as opinions regarding ESMF 
development. 


Observations: 


• No single existing framework can be used without change to construct the ESMF. All frameworks 
incorporate some desirable features; all lack some desirable features. 

• The majority of the climate community code is written in Fortran 77/90. On the other hand, Fortran 
is not extensively used outside the research community. Most researchers do not w^ant to miurate to 
other languages (e.g., C++). The scientific community questions C++ performance relative to 
Fortran. 

The most powerful frameworks identified (ROOT and Overture) are written in C++. No equivalently 
power! ul frameworks are written in Fortran. There are, however, many useful Fortran libraries and 
utilities. 

• Fortran is missing two key OO features that arc available in C++ and Java; inheritance and 
polymorphism. Fortran 2000 is scheduled to have these features in a few years. 

• C++ be combined with Fortran in only a limited w r ay. There is no standard wav for C++ 
programs to read dynamically -allocated Fortran 90 data structures. This problem will still exist with 
Fortran 2000. 

Opinions: 

• OO design and algorithmic design arc complements to one another, not competitors. Some system 
components arc better expressed with algorithms. Other system components are better expressed 
with objects. 

• Most complex systems incorporate complementary components. Microsoft Excel is both data visible 
(spreadsheet) and algorithm visible (Visual Basic Macros). The Overture framework has a high-level 
OO structure but some ol its linear solvers are written in Fortran. 
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• Physics focuses on algorithmic expressions, which are best represented by a lunelional language 
such as Fortran. Frameworks locus on relationships between participating objects (models), which 
are best represented by an OO language, such as C++, Java, or Smalltalk. 

• It may be that in the climate community, the issue ol language, instead ol evolving properly into a 
question regarding the appropriate blend ol complements (OO and algorithmic cxpicssion. Java and 
Fortran, etc.) has evolved into a competition between complements (Fortran or C++, etc. ) 

• The issue of language is absolutely critical to the design of the ESMF 

• If a community ignores the complementary relationship and tries to design a framcwoik using a 
functional language (Fortran) instead ol an OO language, it will incut complexity pioblcms in the 
complementary space. For example, POOMAs goal is to achieve Foitran pcrtoimancc with an OO 
language (C++). It uses expression templates to make compiler optimizations for which it was not 
designed. The result is extreme complexity. On the other hand, if a framework is designed using 
Fortran, the framework will be less capable, more complex, and will have less llcxibility in the 
interfaces. 

• OO design is critical for building a flexible and extensible ESMF meeting the CAN requirements. 
Inheritance and polymorphism are the important OO features that are building blocks of a 
comprehensive framework. Inheritance is a way ol distributing capabilities to models that minimizes 
the programming burden on the researchers. Polymorphism provides a simple mechanism loi 
researchers to modify models. Together, inheritance and polymorphism reduce cost and error. They 
provide a clear hierarchical structure for a complex problem. 

• Properties of a framework are derived from the capabilities of the underlying programming 
languages. A consensus and commitment with respect to language arc crucial lor developing ESMF" 
in the long run. 

• A practical solution must balance cultural and technical factors. It must balance the high value ol 
legacy Fortran code and Fortran expertise with the llcxibility and extensibility of OO languages and 
their widespread use in industry. 

• A practical solution must blend OO and functional design approaches: OO design and flexibility for 
the f ramework and functional design and perlormance lor the physics. 

• An ideal framework would blend the underlying OO architecture ol ROOT and/or Overture, the 
developmental and language scope of Cactus, the optimization approach of Overture, the coupling 
design of DDB, the performance of Fortran 90, the documentation and configuration management ol 
Flux Coupler, and the utilities of GEMS and FMS. 

• If Cactus did have climate-ready features immediately available it is almost certain that the climate 
community would want to use it, even il it meant learning something new (Cactus system and control 
words) and making modifications to the upper layers of their model soltware in Fortran. However 
they must be willing to learn something new and make some changes to their software. II the climate 
community is unwilling or unable to migrate to an OO language lor the framework, then Cactus may 
be a good choice. 
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